[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:25 dist-test-slave-b357 krb5kdc[2183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:25 dist-test-slave-b357 krb5kdc[2183](info): set up 2 sockets
Jun 21 05:20:25 dist-test-slave-b357 krb5kdc[2183](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260621 05:20:26.188094 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:20:26 dist-test-slave-b357 krb5kdc[2183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:20:26.312791 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:44529
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:43945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:44529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:26.399466 2196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:26.399753 2196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:26.399782 2196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:26.401405 2196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:26.401475 2196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:26.401490 2196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:26.401504 2196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:26.401517 2196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:26.403467 2196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:43945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:44529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:44529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:26.403798 2196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:26.404083 2196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:26.409921 2204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:26.409920 2202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:26.409936 2201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:26.410480 2196 server_base.cc:1034] running on GCE node
I20260621 05:20:26.410761 2196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:26.411166 2196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:26.412384 2196 hybrid_clock.cc:648] HybridClock initialized: now 1782019226412343 us; error 69 us; skew 500 ppm
Jun 21 05:20:26 dist-test-slave-b357 krb5kdc[2183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019226, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:26.415205 2196 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:20:26.415714 2196 webserver.cc:460] Webserver started at http://127.2.32.62:39699/ using document root <none> and password file <none>
I20260621 05:20:26.416066 2196 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:26.416129 2196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:26.416311 2196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:26.417450 2196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "7a5b6f70751440f4a1f13b51612e06e6"
format_stamp: "Formatted at 2026-06-21 05:20:26 on dist-test-slave-b357"
server_key: "09d890c982064a41f4a65379b9f15e06"
server_key_iv: "52f49f6c8b52c3725714827e2bc9215c"
server_key_version: "encryptionkey@0"
I20260621 05:20:26.417824 2196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "7a5b6f70751440f4a1f13b51612e06e6"
format_stamp: "Formatted at 2026-06-21 05:20:26 on dist-test-slave-b357"
server_key: "09d890c982064a41f4a65379b9f15e06"
server_key_iv: "52f49f6c8b52c3725714827e2bc9215c"
server_key_version: "encryptionkey@0"
I20260621 05:20:26.419696 2196 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260621 05:20:26.420830 2211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:26.421083 2196 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:26.421207 2196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "7a5b6f70751440f4a1f13b51612e06e6"
format_stamp: "Formatted at 2026-06-21 05:20:26 on dist-test-slave-b357"
server_key: "09d890c982064a41f4a65379b9f15e06"
server_key_iv: "52f49f6c8b52c3725714827e2bc9215c"
server_key_version: "encryptionkey@0"
I20260621 05:20:26.421293 2196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:26.435313 2196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:26.437605 2196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:26.437769 2196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:26.442262 2196 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:44529
I20260621 05:20:26.442525 2263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:44529 every 8 connection(s)
I20260621 05:20:26.442699 2196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:26.443897 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:26.446842 2264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6: Bootstrap starting.
I20260621 05:20:26.447641 2264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:26.448000 2264 log.cc:824] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:26.448297 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 2196
I20260621 05:20:26.448426 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:20:26.449062 2176 external_mini_cluster.cc:1428] Setting key 23f2bae3a82c606bde8c795393db742c
I20260621 05:20:26.449177 2264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6: No bootstrap required, opened a new log
I20260621 05:20:26.452288 2264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5b6f70751440f4a1f13b51612e06e6" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 44529 } }
I20260621 05:20:26.452507 2264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:26.452544 2264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5b6f70751440f4a1f13b51612e06e6, State: Initialized, Role: FOLLOWER
I20260621 05:20:26.452710 2264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5b6f70751440f4a1f13b51612e06e6" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 44529 } }
I20260621 05:20:26.452802 2264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:26.452836 2264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:26.452886 2264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:26.454238 2264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5b6f70751440f4a1f13b51612e06e6" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 44529 } }
I20260621 05:20:26.454357 2264 leader_election.cc:302] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a5b6f70751440f4a1f13b51612e06e6; no voters:
I20260621 05:20:26.454520 2264 leader_election.cc:288] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:26.454587 2269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:26.454773 2269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6 [term 1 LEADER]: Becoming Leader. State: Replica: 7a5b6f70751440f4a1f13b51612e06e6, State: Running, Role: LEADER
I20260621 05:20:26.454785 2264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:26.454921 2269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5b6f70751440f4a1f13b51612e06e6" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 44529 } }
I20260621 05:20:26.455412 2270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a5b6f70751440f4a1f13b51612e06e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5b6f70751440f4a1f13b51612e06e6" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 44529 } } }
I20260621 05:20:26.455494 2270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:26.455706 2271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a5b6f70751440f4a1f13b51612e06e6. Latest consensus state: current_term: 1 leader_uuid: "7a5b6f70751440f4a1f13b51612e06e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5b6f70751440f4a1f13b51612e06e6" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 44529 } } }
I20260621 05:20:26.456029 2271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:26.456212 2276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:26.456821 2276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:26.458465 2276 catalog_manager.cc:1347] Generated new cluster ID: e15371cb6f38444dbac487f1121e915b
I20260621 05:20:26.458518 2276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:20:26 dist-test-slave-b357 krb5kdc[2183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:26.469333 2268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:26.450106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58700 (local address 127.2.32.62:44529)
0621 05:20:26.450327 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:26.450335 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:26.450619 (+ 284us) server_negotiation.cc:407] Connection header received
0621 05:20:26.453530 (+ 2911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:26.453536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:26.453581 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:26.453695 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:26.456023 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:26.457467 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:26.458895 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:26.459549 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:26.459667 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:26.464520 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:26.464523 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:26.464524 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:26.464536 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:26.466683 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:26.467408 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:26.467411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:26.467411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:26.467539 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:26.467880 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:26.467881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:26.467881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:26.468203 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:26.468344 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:26.468735 (+ 391us) server_negotiation.cc:299] Negotiation successful
0621 05:20:26.468890 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":102,"threads_started":1}
I20260621 05:20:26.473949 2276 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:26.474498 2276 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:26.487020 2276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a5b6f70751440f4a1f13b51612e06e6: Generated new TSK 0
I20260621 05:20:26.487277 2276 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
I20260621 05:20:26.518290 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:44529
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:43945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:26.608031 2290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:26.608314 2290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:26.608348 2290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:26.610191 2290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:26.610252 2290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:26.610296 2290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:20:26.612248 2290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:43945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:44529
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:26.612581 2290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:26.612890 2290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:26.618537 2296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:26.618546 2295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:26.618530 2298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:26.618806 2290 server_base.cc:1034] running on GCE node
I20260621 05:20:26.619031 2290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:26.619333 2290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:26.620522 2290 hybrid_clock.cc:648] HybridClock initialized: now 1782019226620477 us; error 55 us; skew 500 ppm
Jun 21 05:20:26 dist-test-slave-b357 krb5kdc[2183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019226, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:26.624099 2290 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:20:26.624787 2290 webserver.cc:460] Webserver started at http://127.2.32.1:40893/ using document root <none> and password file <none>
I20260621 05:20:26.625051 2290 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:26.625123 2290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:26.625317 2290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:26.626571 2290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "4581445247d549cd9336266a6d734335"
format_stamp: "Formatted at 2026-06-21 05:20:26 on dist-test-slave-b357"
server_key: "b29765fd5196ff2e57603acb2f57dd6f"
server_key_iv: "d3e5b1a3ba97480f88a38017de2b7566"
server_key_version: "encryptionkey@0"
I20260621 05:20:26.626986 2290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "4581445247d549cd9336266a6d734335"
format_stamp: "Formatted at 2026-06-21 05:20:26 on dist-test-slave-b357"
server_key: "b29765fd5196ff2e57603acb2f57dd6f"
server_key_iv: "d3e5b1a3ba97480f88a38017de2b7566"
server_key_version: "encryptionkey@0"
I20260621 05:20:26.628824 2290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:26.629856 2305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:26.630149 2290 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:26.630290 2290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "4581445247d549cd9336266a6d734335"
format_stamp: "Formatted at 2026-06-21 05:20:26 on dist-test-slave-b357"
server_key: "b29765fd5196ff2e57603acb2f57dd6f"
server_key_iv: "d3e5b1a3ba97480f88a38017de2b7566"
server_key_version: "encryptionkey@0"
I20260621 05:20:26.630395 2290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:26.650727 2290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:26.653607 2290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:26.653800 2290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:26.654083 2290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:26.654452 2290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:26.654501 2290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:26.654543 2290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:26.654570 2290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:26.661773 2290 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:46879
I20260621 05:20:26.661837 2418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:46879 every 8 connection(s)
I20260621 05:20:26.662148 2290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:20:26.665162 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 2290
I20260621 05:20:26.665355 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:20:26.665544 2176 external_mini_cluster.cc:1428] Setting key 98bd4fd77bbcd5047d4a10e1057df745
Jun 21 05:20:26 dist-test-slave-b357 krb5kdc[2183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019226, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:26.673552 2268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:26.662968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:35009 (local address 127.2.32.62:44529)
0621 05:20:26.663044 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:26.663046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:26.663229 (+ 183us) server_negotiation.cc:407] Connection header received
0621 05:20:26.663811 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:26.663813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:26.663835 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:26.663928 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:26.664989 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:26.665922 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:26.667538 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:26.667882 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:26.667983 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:26.670466 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:26.670469 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:26.670470 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:26.670479 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:26.672018 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:26.672473 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:26.672475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:26.672475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:26.672541 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:26.672804 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:26.672806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:26.672806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:26.672946 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:26.673026 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:26.673328 (+ 302us) server_negotiation.cc:299] Negotiation successful
0621 05:20:26.673452 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 05:20:26.673733 2421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:26.662902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:44529 (local address 127.2.32.1:35009)
0621 05:20:26.663088 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:26.663150 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:26.663661 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:26.664052 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:26.664055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:26.664613 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:26.664806 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:26.664810 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:26.666105 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:26.666107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:26.667434 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:26.667438 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:26.668064 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:26.668066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:26.668269 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:26.668856 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:26.668864 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:26.670317 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:26.672131 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:26.672138 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:26.672141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:26.672374 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:26.672615 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:26.672616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:26.672617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:26.672733 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:26.673030 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:26.673037 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:26.673180 (+ 143us) client_negotiation.cc:769] Sending connection context
0621 05:20:26.673288 (+ 108us) client_negotiation.cc:240] Negotiation successful
0621 05:20:26.673487 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":111,"threads_started":1}
I20260621 05:20:26.674280 2419 heartbeater.cc:347] Connected to a master server at 127.2.32.62:44529
I20260621 05:20:26.674392 2419 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:26.674604 2419 heartbeater.cc:511] Master 127.2.32.62:44529 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.32.2@KRBTEST.COM; defaulting to no policy
I20260621 05:20:26.675256 2228 ts_manager.cc:194] Registered new tserver with Master: 4581445247d549cd9336266a6d734335 (127.2.32.1:46879)
I20260621 05:20:26.675925 2228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:35009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
I20260621 05:20:26.696221 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:44529
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:43945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:26.779232 2424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:26.779443 2424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:26.779469 2424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:26.780561 2424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:26.780601 2424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:26.780627 2424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:20:26.782087 2424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:43945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:44529
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:26.782289 2424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:26.782452 2424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:26.787492 2432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:26.787870 2429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:26.787784 2430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:26.788097 2424 server_base.cc:1034] running on GCE node
I20260621 05:20:26.788446 2424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:26.788733 2424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:26.789932 2424 hybrid_clock.cc:648] HybridClock initialized: now 1782019226789920 us; error 38 us; skew 500 ppm
Jun 21 05:20:26 dist-test-slave-b357 krb5kdc[2183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019226, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:26.793185 2424 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:20:26.794082 2424 webserver.cc:460] Webserver started at http://127.2.32.2:40881/ using document root <none> and password file <none>
I20260621 05:20:26.794348 2424 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:26.794420 2424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:26.794579 2424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:26.795706 2424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "89842bb4b7b24ebd8d891c369e94dd44"
format_stamp: "Formatted at 2026-06-21 05:20:26 on dist-test-slave-b357"
server_key: "9e1d86a05fd42fa03220961989f3870e"
server_key_iv: "309cff03e0c2cf40ec598e8a4e6c9bf1"
server_key_version: "encryptionkey@0"
I20260621 05:20:26.796087 2424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "89842bb4b7b24ebd8d891c369e94dd44"
format_stamp: "Formatted at 2026-06-21 05:20:26 on dist-test-slave-b357"
server_key: "9e1d86a05fd42fa03220961989f3870e"
server_key_iv: "309cff03e0c2cf40ec598e8a4e6c9bf1"
server_key_version: "encryptionkey@0"
I20260621 05:20:26.797485 2424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:20:26.798522 2439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:26.798924 2424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:26.799109 2424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "89842bb4b7b24ebd8d891c369e94dd44"
format_stamp: "Formatted at 2026-06-21 05:20:26 on dist-test-slave-b357"
server_key: "9e1d86a05fd42fa03220961989f3870e"
server_key_iv: "309cff03e0c2cf40ec598e8a4e6c9bf1"
server_key_version: "encryptionkey@0"
I20260621 05:20:26.799360 2424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:26.818008 2424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:26.820199 2424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:26.820356 2424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:26.820593 2424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:26.820915 2424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:26.820971 2424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:26.821012 2424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:26.821040 2424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:26.829172 2424 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:35881
I20260621 05:20:26.829222 2552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:35881 every 8 connection(s)
I20260621 05:20:26.829608 2424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:20:26.831775 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 2424
I20260621 05:20:26.831892 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:20:26.832137 2176 external_mini_cluster.cc:1428] Setting key b437ac8a75fe058a180abc33a3d9ad24
Jun 21 05:20:26 dist-test-slave-b357 krb5kdc[2183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019226, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.32.3@KRBTEST.COM; defaulting to no policy
I20260621 05:20:26.840541 2268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:26.830465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:55531 (local address 127.2.32.62:44529)
0621 05:20:26.830515 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:26.830517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:26.830752 (+ 235us) server_negotiation.cc:407] Connection header received
0621 05:20:26.831377 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:26.831379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:26.831393 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:26.831489 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:26.832660 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:26.833472 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:26.834583 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:26.834793 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:26.834882 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:26.837057 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:26.837058 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:26.837058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:26.837066 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:26.838512 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:26.838984 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:26.838985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:26.838986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:26.839051 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:26.839348 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:26.839349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:26.839349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:26.839480 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:26.839754 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:26.840221 (+ 467us) server_negotiation.cc:299] Negotiation successful
0621 05:20:26.840443 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:20:26.840761 2555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:26.830380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:44529 (local address 127.2.32.2:55531)
0621 05:20:26.830630 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:26.830666 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:26.831228 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:26.831551 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:26.831554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:26.832258 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:26.832562 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:26.832568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:26.833566 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:26.833567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:26.834477 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:26.834480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:26.834892 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:26.834893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:26.835020 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:26.835605 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:26.835614 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:26.836963 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:26.838635 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:26.838642 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:26.838645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:26.838880 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:26.839157 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:26.839158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:26.839158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:26.839272 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:26.839727 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:26.839739 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:26.840026 (+ 287us) client_negotiation.cc:769] Sending connection context
0621 05:20:26.840171 (+ 145us) client_negotiation.cc:240] Negotiation successful
0621 05:20:26.840511 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":181,"threads_started":1}
I20260621 05:20:26.841317 2553 heartbeater.cc:347] Connected to a master server at 127.2.32.62:44529
I20260621 05:20:26.841411 2553 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:26.841645 2553 heartbeater.cc:511] Master 127.2.32.62:44529 requested a full tablet report, sending...
I20260621 05:20:26.842070 2228 ts_manager.cc:194] Registered new tserver with Master: 89842bb4b7b24ebd8d891c369e94dd44 (127.2.32.2:35881)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.3@KRBTEST.COM" created.
I20260621 05:20:26.842516 2228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:55531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
I20260621 05:20:26.860667 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:44529
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:43945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:26.954744 2558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:26.954998 2558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:26.955029 2558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:26.956552 2558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:26.956601 2558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:26.956663 2558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:20:26.958609 2558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:43945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:44529
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:26.958935 2558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:26.959343 2558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:26.965550 2564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:26.965539 2566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:26.965572 2558 server_base.cc:1034] running on GCE node
W20260621 05:20:26.965554 2563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:26.966126 2558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:26.966410 2558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:26.967643 2558 hybrid_clock.cc:648] HybridClock initialized: now 1782019226967602 us; error 55 us; skew 500 ppm
Jun 21 05:20:26 dist-test-slave-b357 krb5kdc[2183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019226, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:26.975440 2558 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:20:26.975901 2558 webserver.cc:460] Webserver started at http://127.2.32.3:39169/ using document root <none> and password file <none>
I20260621 05:20:26.976153 2558 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:26.976225 2558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:26.976388 2558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:26.977483 2558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "ed1aca1848a248fbbd8632a5518934be"
format_stamp: "Formatted at 2026-06-21 05:20:26 on dist-test-slave-b357"
server_key: "d1e51de5444fa57d2827f037515866b4"
server_key_iv: "35db049f5055839f17ff08515b6dc8de"
server_key_version: "encryptionkey@0"
I20260621 05:20:26.977881 2558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "ed1aca1848a248fbbd8632a5518934be"
format_stamp: "Formatted at 2026-06-21 05:20:26 on dist-test-slave-b357"
server_key: "d1e51de5444fa57d2827f037515866b4"
server_key_iv: "35db049f5055839f17ff08515b6dc8de"
server_key_version: "encryptionkey@0"
I20260621 05:20:26.979538 2558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:20:26.980553 2573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:26.980842 2558 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:26.980953 2558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "ed1aca1848a248fbbd8632a5518934be"
format_stamp: "Formatted at 2026-06-21 05:20:26 on dist-test-slave-b357"
server_key: "d1e51de5444fa57d2827f037515866b4"
server_key_iv: "35db049f5055839f17ff08515b6dc8de"
server_key_version: "encryptionkey@0"
I20260621 05:20:26.981037 2558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:26.992465 2558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:26.995098 2558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:26.995323 2558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:26.995589 2558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:26.995908 2558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:26.995966 2558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:26.996003 2558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:26.996054 2558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:27.002863 2558 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:42515
I20260621 05:20:27.002910 2686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:42515 every 8 connection(s)
I20260621 05:20:27.003412 2558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:20:27.007323 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 2558
I20260621 05:20:27.007417 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:20:27.007613 2176 external_mini_cluster.cc:1428] Setting key fbcf37cf6e658f57020dda1d7b724c9e
Jun 21 05:20:27 dist-test-slave-b357 krb5kdc[2183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019226, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:27.014592 2268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:27.004194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:52673 (local address 127.2.32.62:44529)
0621 05:20:27.004268 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:27.004271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:27.004592 (+ 321us) server_negotiation.cc:407] Connection header received
0621 05:20:27.005253 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:27.005254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:27.005269 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:27.005323 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:27.006492 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:27.007096 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:27.008596 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:27.008950 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:27.009058 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:27.011886 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:27.011888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:27.011889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:27.011897 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:27.013275 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:27.013701 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:27.013702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:27.013702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:27.013753 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:27.013982 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:27.013982 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:27.013983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:27.014096 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:27.014151 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:27.014424 (+ 273us) server_negotiation.cc:299] Negotiation successful
0621 05:20:27.014528 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 05:20:27.014910 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:27.004171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:44529 (local address 127.2.32.3:52673)
0621 05:20:27.004357 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:27.004468 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:27.005125 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:27.005429 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:27.005438 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:27.006124 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:27.006381 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:27.006386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:27.007272 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:27.007274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:27.008495 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:27.008498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:27.009124 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:27.009127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:27.009285 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:27.009919 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:27.009929 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:27.011737 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:27.013366 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:27.013371 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:27.013373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:27.013617 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:27.013816 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:27.013817 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:27.013817 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:27.013909 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:27.014153 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:27.014156 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:27.014267 (+ 111us) client_negotiation.cc:769] Sending connection context
0621 05:20:27.014347 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 05:20:27.014555 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260621 05:20:27.015441 2687 heartbeater.cc:347] Connected to a master server at 127.2.32.62:44529
I20260621 05:20:27.015558 2687 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:27.015758 2687 heartbeater.cc:511] Master 127.2.32.62:44529 requested a full tablet report, sending...
I20260621 05:20:27.016390 2228 ts_manager.cc:194] Registered new tserver with Master: ed1aca1848a248fbbd8632a5518934be (127.2.32.3:42515)
I20260621 05:20:27.016880 2228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:52673
I20260621 05:20:27.020588 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 05:20:27 dist-test-slave-b357 krb5kdc[2183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019227, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 05:20:27 dist-test-slave-b357 krb5kdc[2183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019227, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.32.1@KRBTEST.COM
I20260621 05:20:27.047812 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:27.037640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.2.32.1:46879)
0621 05:20:27.037928 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:27.037933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:27.037951 (+ 18us) server_negotiation.cc:407] Connection header received
0621 05:20:27.037987 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:27.037989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:27.038009 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:27.038118 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:27.038861 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:27.039875 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:27.040696 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:27.041048 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:27.041119 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:27.044255 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:27.044262 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:27.044262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:27.044269 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:27.046140 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:27.046568 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:27.046573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:27.046573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:27.046681 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:27.046955 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:27.046956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:27.046956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:27.047203 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:27.047428 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:27.047542 (+ 114us) server_negotiation.cc:299] Negotiation successful
0621 05:20:27.047632 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":133,"threads_started":1}
W20260621 05:20:27.048230 2333 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44986
Jun 21 05:20:27 dist-test-slave-b357 krb5kdc[2183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 05:20:27 dist-test-slave-b357 krb5kdc[2183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.1@KRBTEST.COM
I20260621 05:20:27.074432 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:27.066128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44994 (local address 127.2.32.1:46879)
0621 05:20:27.066199 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:27.066201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:27.066336 (+ 135us) server_negotiation.cc:407] Connection header received
0621 05:20:27.066391 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:27.066393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:27.066409 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:27.066494 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:27.067299 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:27.067860 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:27.068672 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:27.068919 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:27.068991 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:27.071744 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:27.071745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:27.071746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:27.071760 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:27.073244 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:27.073628 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:27.073630 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:27.073630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:27.073688 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:27.073898 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:27.073898 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:27.073899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:27.073998 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:27.074120 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:27.074220 (+ 100us) server_negotiation.cc:299] Negotiation successful
0621 05:20:27.074311 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 05:20:27.075388 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 2290
I20260621 05:20:27.080369 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 2424
I20260621 05:20:27.085919 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 2558
I20260621 05:20:27.090938 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 2196
2026-06-21T05:20:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1987 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:27 dist-test-slave-b357 krb5kdc[2701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:27 dist-test-slave-b357 krb5kdc[2701](info): set up 2 sockets
Jun 21 05:20:27 dist-test-slave-b357 krb5kdc[2701](info): commencing operation
krb5kdc: starting...
W20260621 05:20:28.143039 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:20:28 dist-test-slave-b357 krb5kdc[2701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:20:28.266842 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:37841
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:44255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:37841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:28.356652 2714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:28.356912 2714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:28.356966 2714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:28.358610 2714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:28.358680 2714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:28.358705 2714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:28.358736 2714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:28.358769 2714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:28.360400 2714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:44255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:37841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:37841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:28.360705 2714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:28.360960 2714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:28.366132 2720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:28.366148 2722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:28.366221 2719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:28.366371 2714 server_base.cc:1034] running on GCE node
I20260621 05:20:28.366667 2714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:28.367008 2714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:28.368193 2714 hybrid_clock.cc:648] HybridClock initialized: now 1782019228368162 us; error 59 us; skew 500 ppm
Jun 21 05:20:28 dist-test-slave-b357 krb5kdc[2701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019228, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:28.371680 2714 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:20:28.372184 2714 webserver.cc:460] Webserver started at http://127.2.32.62:37783/ using document root <none> and password file <none>
I20260621 05:20:28.372409 2714 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:28.372476 2714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:28.372639 2714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:28.373705 2714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "12091e3a86924ee49c5fc7625777f963"
format_stamp: "Formatted at 2026-06-21 05:20:28 on dist-test-slave-b357"
server_key: "2716a8b6692452644a270ee096f29efb"
server_key_iv: "cf5711df13ac7ed0bbeaca861985fd35"
server_key_version: "encryptionkey@0"
I20260621 05:20:28.374083 2714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "12091e3a86924ee49c5fc7625777f963"
format_stamp: "Formatted at 2026-06-21 05:20:28 on dist-test-slave-b357"
server_key: "2716a8b6692452644a270ee096f29efb"
server_key_iv: "cf5711df13ac7ed0bbeaca861985fd35"
server_key_version: "encryptionkey@0"
I20260621 05:20:28.375494 2714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:20:28.376362 2729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:28.376641 2714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:28.376767 2714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "12091e3a86924ee49c5fc7625777f963"
format_stamp: "Formatted at 2026-06-21 05:20:28 on dist-test-slave-b357"
server_key: "2716a8b6692452644a270ee096f29efb"
server_key_iv: "cf5711df13ac7ed0bbeaca861985fd35"
server_key_version: "encryptionkey@0"
I20260621 05:20:28.376847 2714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:28.396739 2714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:28.399575 2714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:28.399735 2714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:28.404042 2714 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:37841
I20260621 05:20:28.404263 2781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:37841 every 8 connection(s)
I20260621 05:20:28.404479 2714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:28.405624 2782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:28.408806 2782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963: Bootstrap starting.
I20260621 05:20:28.409588 2782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:28.409968 2782 log.cc:824] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:28.410952 2782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963: No bootstrap required, opened a new log
I20260621 05:20:28.411886 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 2714
I20260621 05:20:28.411968 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:20:28.412165 2176 external_mini_cluster.cc:1428] Setting key 0d3c829c430e784e600d24cabcd8b4d1
I20260621 05:20:28.414013 2782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12091e3a86924ee49c5fc7625777f963" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 37841 } }
I20260621 05:20:28.414249 2782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:28.414289 2782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12091e3a86924ee49c5fc7625777f963, State: Initialized, Role: FOLLOWER
I20260621 05:20:28.414441 2782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12091e3a86924ee49c5fc7625777f963" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 37841 } }
I20260621 05:20:28.414536 2782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:28.414583 2782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:28.414633 2782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:28.415696 2782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12091e3a86924ee49c5fc7625777f963" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 37841 } }
I20260621 05:20:28.415814 2782 leader_election.cc:302] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12091e3a86924ee49c5fc7625777f963; no voters:
I20260621 05:20:28.416002 2782 leader_election.cc:288] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:28.416038 2787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:28.416155 2787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963 [term 1 LEADER]: Becoming Leader. State: Replica: 12091e3a86924ee49c5fc7625777f963, State: Running, Role: LEADER
I20260621 05:20:28.416244 2787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12091e3a86924ee49c5fc7625777f963" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 37841 } }
I20260621 05:20:28.416306 2782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:28.416601 2787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12091e3a86924ee49c5fc7625777f963" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12091e3a86924ee49c5fc7625777f963" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 37841 } } }
I20260621 05:20:28.416678 2787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:28.416812 2787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12091e3a86924ee49c5fc7625777f963. Latest consensus state: current_term: 1 leader_uuid: "12091e3a86924ee49c5fc7625777f963" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12091e3a86924ee49c5fc7625777f963" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 37841 } } }
I20260621 05:20:28.416901 2787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:28.417042 2792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:28.417615 2792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 21 05:20:28 dist-test-slave-b357 krb5kdc[2701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:28.420684 2792 catalog_manager.cc:1347] Generated new cluster ID: 08c8a73affb0430090d1c7019281bf07
I20260621 05:20:28.420737 2792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:28.425776 2786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:28.412760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.2.32.62:37841)
0621 05:20:28.412977 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:28.412988 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:28.413027 (+ 39us) server_negotiation.cc:407] Connection header received
0621 05:20:28.414250 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:28.414259 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:28.414330 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:28.414490 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:28.415392 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:28.416846 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:28.417918 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:28.418365 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:28.418441 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:28.421194 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:28.421198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:28.421198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:28.421209 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:28.423753 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:28.424248 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:28.424252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:28.424252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:28.424410 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:28.424652 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:28.424653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:28.424653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:28.425057 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:28.425250 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:28.425342 (+ 92us) server_negotiation.cc:299] Negotiation successful
0621 05:20:28.425508 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":90,"threads_started":1}
I20260621 05:20:28.433977 2792 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:28.434481 2792 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:28.441447 2792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12091e3a86924ee49c5fc7625777f963: Generated new TSK 0
I20260621 05:20:28.441627 2792 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
I20260621 05:20:28.474647 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:37841
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:44255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:28.567963 2808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:28.568183 2808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:28.568224 2808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:28.569422 2808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:28.569484 2808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:28.569535 2808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:20:28.571008 2808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:44255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:37841
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:28.571285 2808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:28.571513 2808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:28.576403 2814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:28.576416 2816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:28.576423 2813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:28.576736 2808 server_base.cc:1034] running on GCE node
I20260621 05:20:28.576960 2808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:28.577188 2808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:28.578375 2808 hybrid_clock.cc:648] HybridClock initialized: now 1782019228578350 us; error 43 us; skew 500 ppm
Jun 21 05:20:28 dist-test-slave-b357 krb5kdc[2701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019228, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:28.581264 2808 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:20:28.581732 2808 webserver.cc:460] Webserver started at http://127.2.32.1:42591/ using document root <none> and password file <none>
I20260621 05:20:28.581971 2808 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:28.582021 2808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:28.582152 2808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:28.583305 2808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "be42e488d2c54188929046ef182e472f"
format_stamp: "Formatted at 2026-06-21 05:20:28 on dist-test-slave-b357"
server_key: "b11d327e01d1843769bc4b2cec6b81ff"
server_key_iv: "8e940f723cdec0b0781b38e8f6c9416a"
server_key_version: "encryptionkey@0"
I20260621 05:20:28.583673 2808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "be42e488d2c54188929046ef182e472f"
format_stamp: "Formatted at 2026-06-21 05:20:28 on dist-test-slave-b357"
server_key: "b11d327e01d1843769bc4b2cec6b81ff"
server_key_iv: "8e940f723cdec0b0781b38e8f6c9416a"
server_key_version: "encryptionkey@0"
I20260621 05:20:28.585094 2808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:28.586002 2823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:28.586364 2808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:28.586488 2808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "be42e488d2c54188929046ef182e472f"
format_stamp: "Formatted at 2026-06-21 05:20:28 on dist-test-slave-b357"
server_key: "b11d327e01d1843769bc4b2cec6b81ff"
server_key_iv: "8e940f723cdec0b0781b38e8f6c9416a"
server_key_version: "encryptionkey@0"
I20260621 05:20:28.586803 2808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:28.618698 2808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:28.621392 2808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:28.621630 2808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:28.621966 2808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:28.622373 2808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:28.622421 2808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:28.622447 2808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:28.622460 2808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:28.629801 2808 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:42743
I20260621 05:20:28.629863 2936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:42743 every 8 connection(s)
I20260621 05:20:28.630237 2808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:20:28 dist-test-slave-b357 krb5kdc[2701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019228, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:28.640386 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 2808
I20260621 05:20:28.640484 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:20:28.640653 2176 external_mini_cluster.cc:1428] Setting key 9b3718542bfbae1d43966106c641abd5
I20260621 05:20:28.641436 2786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:28.631155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:57203 (local address 127.2.32.62:37841)
0621 05:20:28.631274 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:28.631278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:28.631585 (+ 307us) server_negotiation.cc:407] Connection header received
0621 05:20:28.632659 (+ 1074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:28.632660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:28.632676 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:28.632766 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:28.633558 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:28.634090 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:28.635338 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:28.635542 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:28.635605 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:28.638367 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:28.638368 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:28.638368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:28.638374 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:28.639805 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:28.640220 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:28.640221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:28.640222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:28.640287 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:28.640651 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:28.640652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:28.640652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:28.640794 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:28.640874 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:28.641227 (+ 353us) server_negotiation.cc:299] Negotiation successful
0621 05:20:28.641328 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 05:20:28.641605 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:28.631083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:37841 (local address 127.2.32.1:57203)
0621 05:20:28.631344 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:28.631420 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:28.632483 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:28.632912 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:28.632918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:28.633221 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:28.633439 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:28.633444 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:28.634200 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:28.634201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:28.635229 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:28.635232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:28.635626 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:28.635627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:28.635726 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:28.636376 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:28.636384 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:28.638271 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:28.639884 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:28.639889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:28.639890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:28.640128 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:28.640399 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:28.640400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:28.640400 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:28.640592 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:28.640873 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:28.640881 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:28.641066 (+ 185us) client_negotiation.cc:769] Sending connection context
0621 05:20:28.641184 (+ 118us) client_negotiation.cc:240] Negotiation successful
0621 05:20:28.641361 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":138,"threads_started":1}
I20260621 05:20:28.642099 2937 heartbeater.cc:347] Connected to a master server at 127.2.32.62:37841
I20260621 05:20:28.642220 2937 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:28.642395 2937 heartbeater.cc:511] Master 127.2.32.62:37841 requested a full tablet report, sending...
I20260621 05:20:28.642975 2746 ts_manager.cc:194] Registered new tserver with Master: be42e488d2c54188929046ef182e472f (127.2.32.1:42743)
I20260621 05:20:28.643610 2746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:57203
WARNING: no policy specified for kudu/127.2.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
I20260621 05:20:28.666195 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:37841
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:44255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:28.753104 2942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:28.753326 2942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:28.753363 2942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:28.754465 2942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:28.754528 2942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:28.754572 2942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:20:28.756083 2942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:44255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:37841
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:28.756331 2942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:28.756554 2942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:28.762177 2948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:28.762221 2950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:28.762290 2947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:28.762384 2942 server_base.cc:1034] running on GCE node
I20260621 05:20:28.762625 2942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:28.762882 2942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:28.764078 2942 hybrid_clock.cc:648] HybridClock initialized: now 1782019228764093 us; error 119 us; skew 500 ppm
Jun 21 05:20:28 dist-test-slave-b357 krb5kdc[2701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019228, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:28.767010 2942 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:20:28.767551 2942 webserver.cc:460] Webserver started at http://127.2.32.2:35569/ using document root <none> and password file <none>
I20260621 05:20:28.767817 2942 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:28.767877 2942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:28.768011 2942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:28.769600 2942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "a5dd4040b0bf40efbdc1f8538ea05171"
format_stamp: "Formatted at 2026-06-21 05:20:28 on dist-test-slave-b357"
server_key: "744e0af2104735b294bb5bf057aa7639"
server_key_iv: "08ff4900b90b8cfcd4bee574e9022449"
server_key_version: "encryptionkey@0"
I20260621 05:20:28.770056 2942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "a5dd4040b0bf40efbdc1f8538ea05171"
format_stamp: "Formatted at 2026-06-21 05:20:28 on dist-test-slave-b357"
server_key: "744e0af2104735b294bb5bf057aa7639"
server_key_iv: "08ff4900b90b8cfcd4bee574e9022449"
server_key_version: "encryptionkey@0"
I20260621 05:20:28.771756 2942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260621 05:20:28.772749 2957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:28.773011 2942 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:28.773092 2942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "a5dd4040b0bf40efbdc1f8538ea05171"
format_stamp: "Formatted at 2026-06-21 05:20:28 on dist-test-slave-b357"
server_key: "744e0af2104735b294bb5bf057aa7639"
server_key_iv: "08ff4900b90b8cfcd4bee574e9022449"
server_key_version: "encryptionkey@0"
I20260621 05:20:28.773198 2942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:28.788568 2942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:28.791345 2942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:28.791523 2942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:28.791819 2942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:28.792169 2942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:28.792330 2942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:28.792423 2942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:28.792456 2942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:28.798527 2942 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:42929
I20260621 05:20:28.798564 3070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:42929 every 8 connection(s)
I20260621 05:20:28.798904 2942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:20:28.801853 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 2942
I20260621 05:20:28.801970 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:20:28.802184 2176 external_mini_cluster.cc:1428] Setting key 5e6420d83a6d1f98be9171da7d805c13
Jun 21 05:20:28 dist-test-slave-b357 krb5kdc[2701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019228, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.32.3@KRBTEST.COM; defaulting to no policy
I20260621 05:20:28.810832 2786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:28.799761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:53269 (local address 127.2.32.62:37841)
0621 05:20:28.799855 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:28.799858 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:28.800048 (+ 190us) server_negotiation.cc:407] Connection header received
0621 05:20:28.801143 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:28.801145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:28.801159 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:28.801210 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:28.802073 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:28.802804 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:28.804417 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:28.804847 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:28.804943 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:28.807234 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:28.807236 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:28.807237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:28.807244 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:28.809056 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:28.809710 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:28.809711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:28.809712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:28.809773 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:28.810121 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:28.810122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:28.810123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:28.810279 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:28.810363 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:28.810633 (+ 270us) server_negotiation.cc:299] Negotiation successful
0621 05:20:28.810713 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 05:20:28.810904 3073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:28.799649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:37841 (local address 127.2.32.2:53269)
0621 05:20:28.799905 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:28.799957 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:28.801007 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:28.801294 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:28.801298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:28.801588 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:28.801906 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:28.801912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:28.802918 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:28.802919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:28.804291 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:28.804294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:28.804967 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:28.804969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:28.805075 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:28.805658 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:28.805667 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:28.807076 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:28.809240 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:28.809252 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:28.809255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:28.809603 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:28.809888 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:28.809889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:28.809889 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:28.810028 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:28.810371 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:28.810376 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:28.810497 (+ 121us) client_negotiation.cc:769] Sending connection context
0621 05:20:28.810586 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 05:20:28.810711 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":99,"threads_started":1}
I20260621 05:20:28.811477 3071 heartbeater.cc:347] Connected to a master server at 127.2.32.62:37841
I20260621 05:20:28.811658 3071 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:28.811973 3071 heartbeater.cc:511] Master 127.2.32.62:37841 requested a full tablet report, sending...
I20260621 05:20:28.812461 2746 ts_manager.cc:194] Registered new tserver with Master: a5dd4040b0bf40efbdc1f8538ea05171 (127.2.32.2:42929)
I20260621 05:20:28.812844 2746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:53269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
I20260621 05:20:28.829807 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:37841
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:44255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:28.926582 3076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:28.926884 3076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:28.926925 3076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:28.928503 3076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:28.928565 3076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:28.928611 3076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:20:28.930105 3076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:44255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:37841
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:28.930387 3076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:28.930629 3076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:28.935781 3084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:28.935802 3082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:28.935802 3081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:28.935844 3076 server_base.cc:1034] running on GCE node
I20260621 05:20:28.936183 3076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:28.936416 3076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:28.937563 3076 hybrid_clock.cc:648] HybridClock initialized: now 1782019228937543 us; error 37 us; skew 500 ppm
Jun 21 05:20:28 dist-test-slave-b357 krb5kdc[2701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019228, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:28.940241 3076 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:20:28.940717 3076 webserver.cc:460] Webserver started at http://127.2.32.3:42217/ using document root <none> and password file <none>
I20260621 05:20:28.940950 3076 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:28.941013 3076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:28.941175 3076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:28.942309 3076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "8c952284fde5451d967a95a834f170f8"
format_stamp: "Formatted at 2026-06-21 05:20:28 on dist-test-slave-b357"
server_key: "3ba53645b3150237830ba167305fd70a"
server_key_iv: "aafe4577c43bf1e7a8913f9f6e4b6275"
server_key_version: "encryptionkey@0"
I20260621 05:20:28.942608 3076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "8c952284fde5451d967a95a834f170f8"
format_stamp: "Formatted at 2026-06-21 05:20:28 on dist-test-slave-b357"
server_key: "3ba53645b3150237830ba167305fd70a"
server_key_iv: "aafe4577c43bf1e7a8913f9f6e4b6275"
server_key_version: "encryptionkey@0"
I20260621 05:20:28.943979 3076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:20:28.944926 3091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:28.945212 3076 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:28.945367 3076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "8c952284fde5451d967a95a834f170f8"
format_stamp: "Formatted at 2026-06-21 05:20:28 on dist-test-slave-b357"
server_key: "3ba53645b3150237830ba167305fd70a"
server_key_iv: "aafe4577c43bf1e7a8913f9f6e4b6275"
server_key_version: "encryptionkey@0"
I20260621 05:20:28.945472 3076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:28.958340 3076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:28.960934 3076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:28.961127 3076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:28.961427 3076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:28.961782 3076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:28.961835 3076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:28.961879 3076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:28.961910 3076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:28.968380 3076 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:34355
I20260621 05:20:28.968539 3204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:34355 every 8 connection(s)
I20260621 05:20:28.968848 3076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:20:28.972870 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 3076
I20260621 05:20:28.972982 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:20:28.973160 2176 external_mini_cluster.cc:1428] Setting key 118f1c6f993f281da9218b4d1a75fd20
Jun 21 05:20:28 dist-test-slave-b357 krb5kdc[2701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019228, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:28.979736 2786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:28.969721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:42975 (local address 127.2.32.62:37841)
0621 05:20:28.969807 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:28.969809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:28.970022 (+ 213us) server_negotiation.cc:407] Connection header received
0621 05:20:28.970957 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:28.970959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:28.970972 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:28.971060 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:28.971816 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:28.972320 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:28.973729 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:28.973997 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:28.974086 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:28.976459 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:28.976460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:28.976470 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:28.976476 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:28.978066 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:28.978578 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:28.978579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:28.978580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:28.978638 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:28.978918 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:28.978919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:28.978919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:28.979024 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:28.979079 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:28.979607 (+ 528us) server_negotiation.cc:299] Negotiation successful
0621 05:20:28.979674 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 05:20:28.979861 3207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:28.969644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:37841 (local address 127.2.32.3:42975)
0621 05:20:28.969874 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:28.969915 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:28.970821 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:28.971192 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:28.971196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:28.971472 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:28.971662 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:28.971666 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:28.972456 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:28.972457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:28.973650 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:28.973653 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:28.974103 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:28.974104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:28.974234 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:28.974790 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:28.974798 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:28.976340 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:28.978243 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:28.978248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:28.978250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:28.978483 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:28.978723 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:28.978724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:28.978724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:28.978856 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:28.979310 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:28.979322 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:28.979480 (+ 158us) client_negotiation.cc:769] Sending connection context
0621 05:20:28.979587 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 05:20:28.979705 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":96,"threads_started":1}
I20260621 05:20:28.980360 3205 heartbeater.cc:347] Connected to a master server at 127.2.32.62:37841
I20260621 05:20:28.980458 3205 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:28.980604 3205 heartbeater.cc:511] Master 127.2.32.62:37841 requested a full tablet report, sending...
I20260621 05:20:28.980962 2746 ts_manager.cc:194] Registered new tserver with Master: 8c952284fde5451d967a95a834f170f8 (127.2.32.3:34355)
I20260621 05:20:28.981352 2746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:42975
I20260621 05:20:28.985556 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:20:28.995545 2786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:28.987742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.2.32.62:37841)
0621 05:20:28.987861 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:28.987864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:28.987985 (+ 121us) server_negotiation.cc:407] Connection header received
0621 05:20:28.988100 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:28.988101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:28.988113 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:28.988203 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:28.988883 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:28.989498 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:28.990440 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:28.990640 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:28.990720 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:28.992334 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:28.992336 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:28.992336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:28.992345 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:28.993916 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:28.994407 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:28.994409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:28.994409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:28.994479 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:28.994720 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:28.994722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:28.994722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:28.994892 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:28.995020 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:28.995189 (+ 169us) server_negotiation.cc:299] Negotiation successful
0621 05:20:28.995422 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 05:20:28.998134 2746 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:20:28.999948 2746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:20:29.006876 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:29.002859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:42743 (local address 127.0.0.1:54850)
0621 05:20:29.003196 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:29.003211 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:29.003285 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:29.003521 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:29.003525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:29.003529 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:29.003598 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:29.003616 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.004699 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.004700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:29.005674 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:29.005676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.006576 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.006577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:29.006663 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:29.006677 (+ 14us) client_negotiation.cc:769] Sending connection context
0621 05:20:29.006720 (+ 43us) client_negotiation.cc:240] Negotiation successful
0621 05:20:29.006771 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":70,"threads_started":1}
I20260621 05:20:29.007311 3219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:29.003122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54850 (local address 127.2.32.1:42743)
0621 05:20:29.003327 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:29.003330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:29.003339 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:20:29.003391 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:29.003394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:29.003409 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:29.003501 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:29.003686 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.004599 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.005761 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.006480 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.007090 (+ 610us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:29.007185 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:29.007211 (+ 26us) server_negotiation.cc:299] Negotiation successful
0621 05:20:29.007253 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
I20260621 05:20:29.007979 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:29.002839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:42929 (local address 127.0.0.1:47858)
0621 05:20:29.003047 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:29.003084 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:29.003178 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:29.003949 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:29.003950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:29.003952 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:29.004000 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:29.004001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.005382 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.005383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:29.006238 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:29.006240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.007464 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.007466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:29.007559 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:29.007568 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 05:20:29.007605 (+ 37us) client_negotiation.cc:240] Negotiation successful
0621 05:20:29.007670 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":22,"thread_start_us":86,"threads_started":1}
I20260621 05:20:29.007956 2871 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc8f1a836ec4d58a7c82382f72befda (DEFAULT_TABLE table=test-table [id=0878147e372d4d3c9c164a3224b12007]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:29.008358 2871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc8f1a836ec4d58a7c82382f72befda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:29.008383 3216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:29.002881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:34355 (local address 127.0.0.1:59032)
0621 05:20:29.003823 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:29.003835 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:29.003890 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:29.004051 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:29.004051 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:29.004053 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:29.004104 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:29.004105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.005366 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.005367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:29.006742 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:29.006744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.008072 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.008073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:29.008168 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:29.008174 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:20:29.008240 (+ 66us) client_negotiation.cc:240] Negotiation successful
0621 05:20:29.008308 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":262,"thread_start_us":133,"threads_started":1}
I20260621 05:20:29.008471 3221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:29.003635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59032 (local address 127.2.32.3:34355)
0621 05:20:29.003890 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:29.003894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:29.003906 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:20:29.003943 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:29.003945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:29.003959 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:29.004038 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:29.004176 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.005250 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.006919 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.007814 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.007932 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:29.008007 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:29.008310 (+ 303us) server_negotiation.cc:299] Negotiation successful
0621 05:20:29.008392 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":96,"threads_started":1}
I20260621 05:20:29.008606 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:29.003267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47858 (local address 127.2.32.2:42929)
0621 05:20:29.003763 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:29.003768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:29.003778 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:20:29.003815 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:29.003817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:29.003834 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:29.004071 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:29.004160 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.005250 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.006342 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.007303 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.008235 (+ 932us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:29.008375 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:29.008413 (+ 38us) server_negotiation.cc:299] Negotiation successful
0621 05:20:29.008481 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":231,"threads_started":1}
I20260621 05:20:29.009341 3005 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc8f1a836ec4d58a7c82382f72befda (DEFAULT_TABLE table=test-table [id=0878147e372d4d3c9c164a3224b12007]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:29.009655 3005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc8f1a836ec4d58a7c82382f72befda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:29.010974 3222 tablet_bootstrap.cc:492] T 3fc8f1a836ec4d58a7c82382f72befda P be42e488d2c54188929046ef182e472f: Bootstrap starting.
I20260621 05:20:29.011874 3139 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc8f1a836ec4d58a7c82382f72befda (DEFAULT_TABLE table=test-table [id=0878147e372d4d3c9c164a3224b12007]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:29.012260 3139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc8f1a836ec4d58a7c82382f72befda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:29.013185 3223 tablet_bootstrap.cc:492] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171: Bootstrap starting.
I20260621 05:20:29.013736 3223 tablet_bootstrap.cc:654] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:29.014238 3223 log.cc:824] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:29.015025 3223 tablet_bootstrap.cc:492] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171: No bootstrap required, opened a new log
I20260621 05:20:29.015097 3223 ts_tablet_manager.cc:1397] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:20:29.015383 3225 tablet_bootstrap.cc:492] T 3fc8f1a836ec4d58a7c82382f72befda P 8c952284fde5451d967a95a834f170f8: Bootstrap starting.
I20260621 05:20:29.015615 3222 tablet_bootstrap.cc:654] T 3fc8f1a836ec4d58a7c82382f72befda P be42e488d2c54188929046ef182e472f: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:29.016012 3225 tablet_bootstrap.cc:654] T 3fc8f1a836ec4d58a7c82382f72befda P 8c952284fde5451d967a95a834f170f8: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:29.016168 3222 log.cc:824] T 3fc8f1a836ec4d58a7c82382f72befda P be42e488d2c54188929046ef182e472f: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:29.016660 3225 log.cc:824] T 3fc8f1a836ec4d58a7c82382f72befda P 8c952284fde5451d967a95a834f170f8: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:29.016953 3222 tablet_bootstrap.cc:492] T 3fc8f1a836ec4d58a7c82382f72befda P be42e488d2c54188929046ef182e472f: No bootstrap required, opened a new log
I20260621 05:20:29.017024 3222 ts_tablet_manager.cc:1397] T 3fc8f1a836ec4d58a7c82382f72befda P be42e488d2c54188929046ef182e472f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260621 05:20:29.017398 3225 tablet_bootstrap.cc:492] T 3fc8f1a836ec4d58a7c82382f72befda P 8c952284fde5451d967a95a834f170f8: No bootstrap required, opened a new log
I20260621 05:20:29.017472 3225 ts_tablet_manager.cc:1397] T 3fc8f1a836ec4d58a7c82382f72befda P 8c952284fde5451d967a95a834f170f8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:20:29.017400 3223 raft_consensus.cc:348] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dd4040b0bf40efbdc1f8538ea05171" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42929 } } peers { permanent_uuid: "8c952284fde5451d967a95a834f170f8" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34355 } } peers { permanent_uuid: "be42e488d2c54188929046ef182e472f" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 42743 } }
I20260621 05:20:29.017575 3223 raft_consensus.cc:374] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:29.017614 3223 raft_consensus.cc:729] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5dd4040b0bf40efbdc1f8538ea05171, State: Initialized, Role: FOLLOWER
I20260621 05:20:29.017724 3223 consensus_queue.cc:260] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dd4040b0bf40efbdc1f8538ea05171" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42929 } } peers { permanent_uuid: "8c952284fde5451d967a95a834f170f8" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34355 } } peers { permanent_uuid: "be42e488d2c54188929046ef182e472f" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 42743 } }
I20260621 05:20:29.018028 3223 ts_tablet_manager.cc:1428] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:20:29.018157 3071 heartbeater.cc:503] Master 127.2.32.62:37841 was elected leader, sending a full tablet report...
I20260621 05:20:29.018755 3222 raft_consensus.cc:348] T 3fc8f1a836ec4d58a7c82382f72befda P be42e488d2c54188929046ef182e472f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dd4040b0bf40efbdc1f8538ea05171" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42929 } } peers { permanent_uuid: "8c952284fde5451d967a95a834f170f8" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34355 } } peers { permanent_uuid: "be42e488d2c54188929046ef182e472f" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 42743 } }
I20260621 05:20:29.018868 3222 raft_consensus.cc:374] T 3fc8f1a836ec4d58a7c82382f72befda P be42e488d2c54188929046ef182e472f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:29.018887 3222 raft_consensus.cc:729] T 3fc8f1a836ec4d58a7c82382f72befda P be42e488d2c54188929046ef182e472f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be42e488d2c54188929046ef182e472f, State: Initialized, Role: FOLLOWER
I20260621 05:20:29.018996 3222 consensus_queue.cc:260] T 3fc8f1a836ec4d58a7c82382f72befda P be42e488d2c54188929046ef182e472f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dd4040b0bf40efbdc1f8538ea05171" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42929 } } peers { permanent_uuid: "8c952284fde5451d967a95a834f170f8" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34355 } } peers { permanent_uuid: "be42e488d2c54188929046ef182e472f" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 42743 } }
I20260621 05:20:29.019305 2937 heartbeater.cc:503] Master 127.2.32.62:37841 was elected leader, sending a full tablet report...
I20260621 05:20:29.019299 3222 ts_tablet_manager.cc:1428] T 3fc8f1a836ec4d58a7c82382f72befda P be42e488d2c54188929046ef182e472f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260621 05:20:29.019533 3225 raft_consensus.cc:348] T 3fc8f1a836ec4d58a7c82382f72befda P 8c952284fde5451d967a95a834f170f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dd4040b0bf40efbdc1f8538ea05171" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42929 } } peers { permanent_uuid: "8c952284fde5451d967a95a834f170f8" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34355 } } peers { permanent_uuid: "be42e488d2c54188929046ef182e472f" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 42743 } }
I20260621 05:20:29.019706 3225 raft_consensus.cc:374] T 3fc8f1a836ec4d58a7c82382f72befda P 8c952284fde5451d967a95a834f170f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:29.019747 3225 raft_consensus.cc:729] T 3fc8f1a836ec4d58a7c82382f72befda P 8c952284fde5451d967a95a834f170f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c952284fde5451d967a95a834f170f8, State: Initialized, Role: FOLLOWER
I20260621 05:20:29.019887 3225 consensus_queue.cc:260] T 3fc8f1a836ec4d58a7c82382f72befda P 8c952284fde5451d967a95a834f170f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dd4040b0bf40efbdc1f8538ea05171" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42929 } } peers { permanent_uuid: "8c952284fde5451d967a95a834f170f8" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34355 } } peers { permanent_uuid: "be42e488d2c54188929046ef182e472f" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 42743 } }
I20260621 05:20:29.020177 3225 ts_tablet_manager.cc:1428] T 3fc8f1a836ec4d58a7c82382f72befda P 8c952284fde5451d967a95a834f170f8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:20:29.020182 3205 heartbeater.cc:503] Master 127.2.32.62:37841 was elected leader, sending a full tablet report...
W20260621 05:20:29.049573 3072 tablet.cc:2369] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:20:29.074518 3228 raft_consensus.cc:482] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:20:29.074672 3228 raft_consensus.cc:504] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dd4040b0bf40efbdc1f8538ea05171" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42929 } } peers { permanent_uuid: "8c952284fde5451d967a95a834f170f8" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34355 } } peers { permanent_uuid: "be42e488d2c54188929046ef182e472f" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 42743 } }
I20260621 05:20:29.075088 3228 leader_election.cc:288] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c952284fde5451d967a95a834f170f8 (127.2.32.3:34355), be42e488d2c54188929046ef182e472f (127.2.32.1:42743)
I20260621 05:20:29.078635 3219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:29.075391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:44641 (local address 127.2.32.1:42743)
0621 05:20:29.075454 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:29.075456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:29.075573 (+ 117us) server_negotiation.cc:407] Connection header received
0621 05:20:29.075670 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:29.075671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:29.075683 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:29.075755 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:29.075922 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.076393 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.077677 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.078254 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.078336 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:29.078346 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:29.078518 (+ 172us) server_negotiation.cc:299] Negotiation successful
0621 05:20:29.078557 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 05:20:29.078655 3221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:29.075343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:42455 (local address 127.2.32.3:34355)
0621 05:20:29.075415 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:29.075417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:29.075427 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:20:29.075459 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:29.075460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:29.075473 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:29.075535 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:29.075724 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.076196 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.077616 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.078255 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.078336 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:29.078347 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:29.078518 (+ 171us) server_negotiation.cc:299] Negotiation successful
0621 05:20:29.078556 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 05:20:29.078667 3073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:29.075258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:34355 (local address 127.2.32.2:42455)
0621 05:20:29.075341 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:29.075353 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:29.075400 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:29.075563 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:29.075564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:29.075566 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:29.075640 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:29.075642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.076286 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.076287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:29.077514 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:29.077517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.078351 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.078352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:29.078430 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:29.078434 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:20:29.078499 (+ 65us) client_negotiation.cc:240] Negotiation successful
0621 05:20:29.078562 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":4}
I20260621 05:20:29.078675 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:29.075313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:42743 (local address 127.2.32.2:44641)
0621 05:20:29.075499 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:29.075527 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:29.075596 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:29.075758 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:29.075759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:29.075760 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:29.075829 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:29.075831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.076482 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.076483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:29.077587 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:29.077590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.078351 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.078352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:29.078430 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:29.078447 (+ 17us) client_negotiation.cc:769] Sending connection context
0621 05:20:29.078499 (+ 52us) client_negotiation.cc:240] Negotiation successful
0621 05:20:29.078562 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":31,"thread_start_us":124,"threads_started":1}
I20260621 05:20:29.079358 3159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc8f1a836ec4d58a7c82382f72befda" candidate_uuid: "a5dd4040b0bf40efbdc1f8538ea05171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c952284fde5451d967a95a834f170f8" is_pre_election: true
I20260621 05:20:29.079536 3159 raft_consensus.cc:2454] T 3fc8f1a836ec4d58a7c82382f72befda P 8c952284fde5451d967a95a834f170f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5dd4040b0bf40efbdc1f8538ea05171 in term 0.
I20260621 05:20:29.079638 2891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc8f1a836ec4d58a7c82382f72befda" candidate_uuid: "a5dd4040b0bf40efbdc1f8538ea05171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be42e488d2c54188929046ef182e472f" is_pre_election: true
I20260621 05:20:29.079789 2891 raft_consensus.cc:2454] T 3fc8f1a836ec4d58a7c82382f72befda P be42e488d2c54188929046ef182e472f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5dd4040b0bf40efbdc1f8538ea05171 in term 0.
I20260621 05:20:29.079758 2958 leader_election.cc:302] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c952284fde5451d967a95a834f170f8, a5dd4040b0bf40efbdc1f8538ea05171; no voters:
I20260621 05:20:29.079910 3228 raft_consensus.cc:2781] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:20:29.079972 3228 raft_consensus.cc:482] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:20:29.079993 3228 raft_consensus.cc:3037] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:29.080732 3228 raft_consensus.cc:504] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dd4040b0bf40efbdc1f8538ea05171" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42929 } } peers { permanent_uuid: "8c952284fde5451d967a95a834f170f8" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34355 } } peers { permanent_uuid: "be42e488d2c54188929046ef182e472f" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 42743 } }
I20260621 05:20:29.080890 3228 leader_election.cc:288] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171 [CANDIDATE]: Term 1 election: Requested vote from peers 8c952284fde5451d967a95a834f170f8 (127.2.32.3:34355), be42e488d2c54188929046ef182e472f (127.2.32.1:42743)
I20260621 05:20:29.081007 3159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc8f1a836ec4d58a7c82382f72befda" candidate_uuid: "a5dd4040b0bf40efbdc1f8538ea05171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c952284fde5451d967a95a834f170f8"
I20260621 05:20:29.081079 3159 raft_consensus.cc:3037] T 3fc8f1a836ec4d58a7c82382f72befda P 8c952284fde5451d967a95a834f170f8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:29.081105 2891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc8f1a836ec4d58a7c82382f72befda" candidate_uuid: "a5dd4040b0bf40efbdc1f8538ea05171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be42e488d2c54188929046ef182e472f"
I20260621 05:20:29.081178 2891 raft_consensus.cc:3037] T 3fc8f1a836ec4d58a7c82382f72befda P be42e488d2c54188929046ef182e472f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:29.081918 3159 raft_consensus.cc:2454] T 3fc8f1a836ec4d58a7c82382f72befda P 8c952284fde5451d967a95a834f170f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dd4040b0bf40efbdc1f8538ea05171 in term 1.
I20260621 05:20:29.082032 2891 raft_consensus.cc:2454] T 3fc8f1a836ec4d58a7c82382f72befda P be42e488d2c54188929046ef182e472f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dd4040b0bf40efbdc1f8538ea05171 in term 1.
I20260621 05:20:29.082366 2958 leader_election.cc:302] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c952284fde5451d967a95a834f170f8, a5dd4040b0bf40efbdc1f8538ea05171; no voters:
I20260621 05:20:29.082564 3228 raft_consensus.cc:2781] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:29.082810 3228 raft_consensus.cc:686] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171 [term 1 LEADER]: Becoming Leader. State: Replica: a5dd4040b0bf40efbdc1f8538ea05171, State: Running, Role: LEADER
I20260621 05:20:29.082914 3228 consensus_queue.cc:237] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dd4040b0bf40efbdc1f8538ea05171" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42929 } } peers { permanent_uuid: "8c952284fde5451d967a95a834f170f8" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34355 } } peers { permanent_uuid: "be42e488d2c54188929046ef182e472f" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 42743 } }
I20260621 05:20:29.083987 2746 catalog_manager.cc:5495] T 3fc8f1a836ec4d58a7c82382f72befda P a5dd4040b0bf40efbdc1f8538ea05171 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5dd4040b0bf40efbdc1f8538ea05171 (127.2.32.2). New cstate: current_term: 1 leader_uuid: "a5dd4040b0bf40efbdc1f8538ea05171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dd4040b0bf40efbdc1f8538ea05171" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c952284fde5451d967a95a834f170f8" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be42e488d2c54188929046ef182e472f" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 42743 } health_report { overall_health: UNKNOWN } } }
Jun 21 05:20:29 dist-test-slave-b357 krb5kdc[2701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.1@KRBTEST.COM
I20260621 05:20:29.104224 3219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:29.096625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54852 (local address 127.2.32.1:42743)
0621 05:20:29.096710 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:29.096712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:29.096803 (+ 91us) server_negotiation.cc:407] Connection header received
0621 05:20:29.096874 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:29.096875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:29.096886 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:29.096968 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:29.097586 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.098068 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.098784 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.098984 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.099068 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:29.101370 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:29.101380 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:29.101380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:29.101394 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:29.103003 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:29.103426 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:29.103428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:29.103429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:29.103479 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:29.103680 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:29.103681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:29.103682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:29.103817 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:29.103961 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:29.104044 (+ 83us) server_negotiation.cc:299] Negotiation successful
0621 05:20:29.104118 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Jun 21 05:20:29 dist-test-slave-b357 krb5kdc[2701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019229, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 05:20:29 dist-test-slave-b357 krb5kdc[2701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019229, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.32.1@KRBTEST.COM
I20260621 05:20:29.129912 3219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:29.122569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54866 (local address 127.2.32.1:42743)
0621 05:20:29.122635 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:29.122638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:29.122778 (+ 140us) server_negotiation.cc:407] Connection header received
0621 05:20:29.122876 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:29.122877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:29.122887 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:29.122955 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:29.123595 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.124071 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.124793 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.124987 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.125064 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:29.127323 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:29.127324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:29.127325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:29.127332 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:29.128717 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:29.129081 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:29.129082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:29.129082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:29.129139 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:29.129334 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:29.129336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:29.129336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:29.129441 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:29.129543 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:29.129729 (+ 186us) server_negotiation.cc:299] Negotiation successful
0621 05:20:29.129808 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260621 05:20:29.131048 2938 tablet.cc:2369] T 3fc8f1a836ec4d58a7c82382f72befda P be42e488d2c54188929046ef182e472f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 21 05:20:29 dist-test-slave-b357 krb5kdc[2701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 05:20:29 dist-test-slave-b357 krb5kdc[2701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.1@KRBTEST.COM
I20260621 05:20:29.153949 3219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:29.145857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54876 (local address 127.2.32.1:42743)
0621 05:20:29.145949 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:29.145953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:29.146058 (+ 105us) server_negotiation.cc:407] Connection header received
0621 05:20:29.146175 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:29.146176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:29.146189 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:29.146271 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:29.146884 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.147461 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.148306 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:29.148494 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:29.148586 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:29.151436 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:29.151438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:29.151439 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:29.151445 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:29.152823 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:29.153193 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:29.153194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:29.153195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:29.153252 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:29.153450 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:29.153451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:29.153452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:29.153556 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:29.153656 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:29.153771 (+ 115us) server_negotiation.cc:299] Negotiation successful
0621 05:20:29.153843 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260621 05:20:29.154583 2851 tablet_service.cc:2908] Rejecting scan request for tablet 3fc8f1a836ec4d58a7c82382f72befda: Uninitialized: clean time has not yet been initialized
I20260621 05:20:29.155680 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 2808
I20260621 05:20:29.160634 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 2942
I20260621 05:20:29.165530 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 3076
I20260621 05:20:29.171103 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 2714
2026-06-21T05:20:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2079 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:29 dist-test-slave-b357 krb5kdc[3246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:29 dist-test-slave-b357 krb5kdc[3246](info): set up 2 sockets
Jun 21 05:20:29 dist-test-slave-b357 krb5kdc[3246](info): commencing operation
krb5kdc: starting...
W20260621 05:20:30.239919 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:20:30 dist-test-slave-b357 krb5kdc[3246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:20:30.370628 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:42743
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:42743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:30.452872 3259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:30.453130 3259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:30.453163 3259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:30.454267 3259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:30.454330 3259 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 05:20:30.454353 3259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:30.454375 3259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:30.454401 3259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:30.454427 3259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:30.455870 3259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:42743
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:42743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:30.456149 3259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:30.456383 3259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:30.462065 3265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:30.462088 3264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:30.462076 3267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:30.462209 3259 server_base.cc:1034] running on GCE node
I20260621 05:20:30.462707 3259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:30.463058 3259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:30.464241 3259 hybrid_clock.cc:648] HybridClock initialized: now 1782019230464219 us; error 49 us; skew 500 ppm
Jun 21 05:20:30 dist-test-slave-b357 krb5kdc[3246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019230, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:30.466781 3259 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:20:30.467315 3259 webserver.cc:460] Webserver started at http://127.2.32.62:35647/ using document root <none> and password file <none>
I20260621 05:20:30.467504 3259 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:30.467543 3259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:30.467628 3259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:30.468595 3259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "e9ae6a6ba06042b6be64f749919c43bf"
format_stamp: "Formatted at 2026-06-21 05:20:30 on dist-test-slave-b357"
server_key: "7b30d40b04f351ceac0950b7e082cae2"
server_key_iv: "8c6957cd1b38e2e15bb3d86b70112c52"
server_key_version: "encryptionkey@0"
I20260621 05:20:30.469105 3259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "e9ae6a6ba06042b6be64f749919c43bf"
format_stamp: "Formatted at 2026-06-21 05:20:30 on dist-test-slave-b357"
server_key: "7b30d40b04f351ceac0950b7e082cae2"
server_key_iv: "8c6957cd1b38e2e15bb3d86b70112c52"
server_key_version: "encryptionkey@0"
I20260621 05:20:30.470732 3259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:20:30.471805 3274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:30.472115 3259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:20:30.472338 3259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "e9ae6a6ba06042b6be64f749919c43bf"
format_stamp: "Formatted at 2026-06-21 05:20:30 on dist-test-slave-b357"
server_key: "7b30d40b04f351ceac0950b7e082cae2"
server_key_iv: "8c6957cd1b38e2e15bb3d86b70112c52"
server_key_version: "encryptionkey@0"
I20260621 05:20:30.472433 3259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:30.487483 3259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:30.489776 3259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:30.489992 3259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:30.495800 3259 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:42743
I20260621 05:20:30.495849 3336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:42743 every 8 connection(s)
I20260621 05:20:30.496152 3259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:30.496765 3337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:30.499168 3337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf: Bootstrap starting.
I20260621 05:20:30.499861 3337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:30.500198 3337 log.cc:824] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:30.500957 3337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf: No bootstrap required, opened a new log
I20260621 05:20:30.503010 3337 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ae6a6ba06042b6be64f749919c43bf" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 42743 } }
I20260621 05:20:30.503188 3337 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:30.503238 3337 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9ae6a6ba06042b6be64f749919c43bf, State: Initialized, Role: FOLLOWER
I20260621 05:20:30.503381 3337 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ae6a6ba06042b6be64f749919c43bf" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 42743 } }
I20260621 05:20:30.503464 3337 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:30.503508 3337 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:30.503587 3337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:30.504098 3337 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ae6a6ba06042b6be64f749919c43bf" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 42743 } }
I20260621 05:20:30.504215 3337 leader_election.cc:302] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9ae6a6ba06042b6be64f749919c43bf; no voters:
I20260621 05:20:30.504383 3337 leader_election.cc:288] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:30.504527 3340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:30.504647 3337 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:30.504781 3340 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf [term 1 LEADER]: Becoming Leader. State: Replica: e9ae6a6ba06042b6be64f749919c43bf, State: Running, Role: LEADER
I20260621 05:20:30.504921 3340 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ae6a6ba06042b6be64f749919c43bf" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 42743 } }
I20260621 05:20:30.505406 3342 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9ae6a6ba06042b6be64f749919c43bf. Latest consensus state: current_term: 1 leader_uuid: "e9ae6a6ba06042b6be64f749919c43bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ae6a6ba06042b6be64f749919c43bf" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 42743 } } }
I20260621 05:20:30.505527 3342 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:30.505808 3341 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9ae6a6ba06042b6be64f749919c43bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ae6a6ba06042b6be64f749919c43bf" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 42743 } } }
I20260621 05:20:30.505892 3341 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:30.505885 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 3259
I20260621 05:20:30.505991 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:20:30.506172 2176 external_mini_cluster.cc:1428] Setting key 511afe212ed97be486237a9dcaa8e0c8
I20260621 05:20:30.506271 3349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:30.506947 3349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:30.508524 3349 catalog_manager.cc:1347] Generated new cluster ID: 41e1390f52b34440a0d8f0da65125f45
I20260621 05:20:30.508585 3349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:20:30 dist-test-slave-b357 krb5kdc[3246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:30.518664 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:30.506714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59922 (local address 127.2.32.62:42743)
0621 05:20:30.506949 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:30.506959 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:30.506992 (+ 33us) server_negotiation.cc:407] Connection header received
0621 05:20:30.507118 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:30.507149 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:30.507207 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:30.507385 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:30.508309 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:30.509676 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:30.510909 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:30.511727 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:30.512193 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:30.514859 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:30.514863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:30.514864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:30.514875 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:30.517003 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:30.517416 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:30.517419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:30.517419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:30.517515 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:30.517723 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:30.517723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:30.517724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:30.518014 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:30.518158 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:30.518300 (+ 142us) server_negotiation.cc:299] Negotiation successful
0621 05:20:30.518450 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":110,"threads_started":1}
I20260621 05:20:30.527107 3349 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:30.527616 3349 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:30.535923 3349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf: Generated new TSK 0
I20260621 05:20:30.536106 3349 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
I20260621 05:20:30.570297 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:42743
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:30.660305 3363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:30.660519 3363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:30.660543 3363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:30.661690 3363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:30.661732 3363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:30.661753 3363 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:20:30.661772 3363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:20:30.663223 3363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.32.62:42743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:30.663462 3363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:30.663717 3363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:30.668797 3369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:30.668835 3371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:30.668797 3368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:30.669140 3363 server_base.cc:1034] running on GCE node
I20260621 05:20:30.669337 3363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:30.669565 3363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:30.670751 3363 hybrid_clock.cc:648] HybridClock initialized: now 1782019230670737 us; error 59 us; skew 500 ppm
Jun 21 05:20:30 dist-test-slave-b357 krb5kdc[3246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019230, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:30.673328 3363 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:20:30.673756 3363 webserver.cc:460] Webserver started at http://127.2.32.1:43851/ using document root <none> and password file <none>
I20260621 05:20:30.673981 3363 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:30.674049 3363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:30.674197 3363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:30.675310 3363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "161d3b6a40d44954a10ffed41a6f0a16"
format_stamp: "Formatted at 2026-06-21 05:20:30 on dist-test-slave-b357"
server_key: "2321ded446b646e1973c5fa3b5c4f971"
server_key_iv: "ea2a494501df28c42307b994d32ada4b"
server_key_version: "encryptionkey@0"
I20260621 05:20:30.675669 3363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "161d3b6a40d44954a10ffed41a6f0a16"
format_stamp: "Formatted at 2026-06-21 05:20:30 on dist-test-slave-b357"
server_key: "2321ded446b646e1973c5fa3b5c4f971"
server_key_iv: "ea2a494501df28c42307b994d32ada4b"
server_key_version: "encryptionkey@0"
I20260621 05:20:30.677084 3363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:30.678191 3378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:30.678457 3363 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:30.678669 3363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "161d3b6a40d44954a10ffed41a6f0a16"
format_stamp: "Formatted at 2026-06-21 05:20:30 on dist-test-slave-b357"
server_key: "2321ded446b646e1973c5fa3b5c4f971"
server_key_iv: "ea2a494501df28c42307b994d32ada4b"
server_key_version: "encryptionkey@0"
I20260621 05:20:30.678785 3363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:30.686635 3363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:30.689054 3363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:30.689227 3363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:30.689916 3363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:30.689978 3363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:30.690033 3363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:30.690101 3363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:30.697080 3363 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:37679
I20260621 05:20:30.697176 3493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:37679 every 8 connection(s)
Jun 21 05:20:30 dist-test-slave-b357 krb5kdc[3246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019230, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:30.697532 3363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:20:30.701511 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:30.690691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:44937 (local address 127.2.32.62:42743)
0621 05:20:30.690773 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:30.690775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:30.690957 (+ 182us) server_negotiation.cc:407] Connection header received
0621 05:20:30.691545 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:30.691546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:30.691562 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:30.691620 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:30.692489 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:30.693361 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:30.694926 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:30.695293 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:30.695381 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:30.698071 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:30.698082 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:30.698082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:30.698090 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:30.699780 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:30.700462 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:30.700463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:30.700463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:30.700520 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:30.700854 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:30.700855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:30.700855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:30.700990 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:30.701062 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:30.701363 (+ 301us) server_negotiation.cc:299] Negotiation successful
0621 05:20:30.701446 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:20:30.701651 3387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:30.690635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:42743 (local address 127.2.32.1:44937)
0621 05:20:30.690840 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:30.690873 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:30.691438 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:30.691697 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:30.691701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:30.692084 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:30.692375 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:30.692381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:30.693459 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:30.693460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:30.694831 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:30.694834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:30.695383 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:30.695383 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:30.695527 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:30.696103 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:30.696115 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:30.697973 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:30.699928 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:30.699940 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:30.699942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:30.700314 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:30.700627 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:30.700629 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:30.700630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:30.700761 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:30.701096 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:30.701102 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:30.701219 (+ 117us) client_negotiation.cc:769] Sending connection context
0621 05:20:30.701311 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 05:20:30.701444 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":105,"threads_started":1}
I20260621 05:20:30.702232 3494 heartbeater.cc:347] Connected to a master server at 127.2.32.62:42743
I20260621 05:20:30.702333 3494 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:30.702508 3494 heartbeater.cc:511] Master 127.2.32.62:42743 requested a full tablet report, sending...
I20260621 05:20:30.703004 3290 ts_manager.cc:194] Registered new tserver with Master: 161d3b6a40d44954a10ffed41a6f0a16 (127.2.32.1:37679)
I20260621 05:20:30.703713 3290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:44937
I20260621 05:20:30.705782 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 3363
I20260621 05:20:30.705896 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:20:30.706086 2176 external_mini_cluster.cc:1428] Setting key 090bf4fe6c9c6ccbbd1675899feed35b
I20260621 05:20:30.709946 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:30.703253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:42743 (local address 127.2.32.1:56297)
0621 05:20:30.703378 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:30.703389 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:30.703445 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:30.703569 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:30.703570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:30.703744 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:30.703787 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:30.703788 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:30.704715 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:30.704716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:30.705202 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:30.705204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:30.706012 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:30.706013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:30.706095 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:30.706479 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:30.706483 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:30.706805 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:30.708819 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:30.708821 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:30.708821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:30.709025 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:30.709296 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:30.709297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:30.709297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:30.709366 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:30.709680 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:30.709681 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:30.709724 (+ 43us) client_negotiation.cc:769] Sending connection context
0621 05:20:30.709792 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 05:20:30.709873 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260621 05:20:30.710119 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:30.703316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:56297 (local address 127.2.32.62:42743)
0621 05:20:30.703379 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:30.703381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:30.703432 (+ 51us) server_negotiation.cc:407] Connection header received
0621 05:20:30.703509 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:30.703510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:30.703519 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:30.703567 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:30.703865 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:30.704571 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:30.705347 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:30.705874 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:30.706015 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:30.706981 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:30.706982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:30.706983 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:30.706990 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:30.708666 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:30.709106 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:30.709107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:30.709108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:30.709218 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:30.709433 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:30.709434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:30.709434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:30.709591 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:30.709663 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:30.709841 (+ 178us) server_negotiation.cc:299] Negotiation successful
0621 05:20:30.709946 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.2.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
I20260621 05:20:30.734869 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:42743
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:30.818878 3503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:30.819077 3503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:30.819101 3503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:30.820287 3503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:30.820339 3503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:30.820360 3503 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:20:30.820377 3503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:20:30.821776 3503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.32.62:42743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:30.822003 3503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:30.822261 3503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:30.827310 3508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:30.827409 3511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:30.827549 3503 server_base.cc:1034] running on GCE node
W20260621 05:20:30.827270 3509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:30.827843 3503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:30.828095 3503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:30.829293 3503 hybrid_clock.cc:648] HybridClock initialized: now 1782019230829228 us; error 71 us; skew 500 ppm
Jun 21 05:20:30 dist-test-slave-b357 krb5kdc[3246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019230, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:30.832418 3503 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:20:30.832863 3503 webserver.cc:460] Webserver started at http://127.2.32.2:41643/ using document root <none> and password file <none>
I20260621 05:20:30.833091 3503 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:30.833160 3503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:30.833312 3503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:30.834371 3503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "347c0427a88e45c5b1fad396bbe0f26a"
format_stamp: "Formatted at 2026-06-21 05:20:30 on dist-test-slave-b357"
server_key: "0bfbcd200f6c8fedf926ee9bfbf1285f"
server_key_iv: "80145e96584556ed02e378ade7ef402b"
server_key_version: "encryptionkey@0"
I20260621 05:20:30.834736 3503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "347c0427a88e45c5b1fad396bbe0f26a"
format_stamp: "Formatted at 2026-06-21 05:20:30 on dist-test-slave-b357"
server_key: "0bfbcd200f6c8fedf926ee9bfbf1285f"
server_key_iv: "80145e96584556ed02e378ade7ef402b"
server_key_version: "encryptionkey@0"
I20260621 05:20:30.836212 3503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:20:30.837095 3518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:30.837363 3503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:30.837479 3503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "347c0427a88e45c5b1fad396bbe0f26a"
format_stamp: "Formatted at 2026-06-21 05:20:30 on dist-test-slave-b357"
server_key: "0bfbcd200f6c8fedf926ee9bfbf1285f"
server_key_iv: "80145e96584556ed02e378ade7ef402b"
server_key_version: "encryptionkey@0"
I20260621 05:20:30.837565 3503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:30.847863 3503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:30.850611 3503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:30.850831 3503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:30.851689 3503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:30.851785 3503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:30.851830 3503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:30.851853 3503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 05:20:30 dist-test-slave-b357 krb5kdc[3246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019230, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:30.859467 3503 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:41747
I20260621 05:20:30.859568 3633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:41747 every 8 connection(s)
I20260621 05:20:30.859889 3503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:20:30.860138 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 3503
I20260621 05:20:30.860239 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:20:30.860419 2176 external_mini_cluster.cc:1428] Setting key 21d1e70a2546a5c7d30cc4b1d1db0275
I20260621 05:20:30.863512 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:30.852374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:35095 (local address 127.2.32.62:42743)
0621 05:20:30.852438 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:30.852440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:30.852768 (+ 328us) server_negotiation.cc:407] Connection header received
0621 05:20:30.853439 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:30.853442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:30.853465 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:30.853656 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:30.854451 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:30.855433 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:30.856651 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:30.856879 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:30.856962 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:30.859635 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:30.859637 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:30.859637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:30.859646 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:30.861770 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:30.862333 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:30.862334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:30.862335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:30.862387 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:30.862719 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:30.862720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:30.862721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:30.862873 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:30.862956 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:30.863307 (+ 351us) server_negotiation.cc:299] Negotiation successful
0621 05:20:30.863394 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:20:30.863600 3527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:30.852430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:42743 (local address 127.2.32.2:35095)
0621 05:20:30.852640 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:30.852675 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:30.853211 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:30.853639 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:30.853643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:30.854064 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:30.854337 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:30.854360 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:30.855577 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:30.855579 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:30.856539 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:30.856542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:30.856998 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:30.857000 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:30.857136 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:30.857798 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:30.857808 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:30.859531 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:30.861860 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:30.861867 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:30.861870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:30.862238 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:30.862491 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:30.862493 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:30.862493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:30.862647 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:30.862969 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:30.862974 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:30.863096 (+ 122us) client_negotiation.cc:769] Sending connection context
0621 05:20:30.863272 (+ 176us) client_negotiation.cc:240] Negotiation successful
0621 05:20:30.863421 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":116,"threads_started":1}
I20260621 05:20:30.864048 3634 heartbeater.cc:347] Connected to a master server at 127.2.32.62:42743
I20260621 05:20:30.864145 3634 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:30.864312 3634 heartbeater.cc:511] Master 127.2.32.62:42743 requested a full tablet report, sending...
I20260621 05:20:30.864666 3290 ts_manager.cc:194] Registered new tserver with Master: 347c0427a88e45c5b1fad396bbe0f26a (127.2.32.2:41747)
I20260621 05:20:30.865113 3290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:35095
WARNING: no policy specified for kudu/127.2.32.3@KRBTEST.COM; defaulting to no policy
I20260621 05:20:30.872083 3641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:30.865206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:42743 (local address 127.2.32.2:43089)
0621 05:20:30.865367 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:30.865379 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:30.865459 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:30.865629 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:30.865630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:30.865854 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:30.865954 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:30.865956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:30.866641 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:30.866642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:30.867552 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:30.867555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:30.868016 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:30.868018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:30.868102 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:30.868494 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:30.868500 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:30.868923 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:30.870662 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:30.870663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:30.870663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:30.870820 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:30.871276 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:30.871278 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:30.871279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:30.871364 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:30.871779 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:30.871780 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:30.871840 (+ 60us) client_negotiation.cc:769] Sending connection context
0621 05:20:30.871921 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 05:20:30.872016 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":68,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.3@KRBTEST.COM" created.
I20260621 05:20:30.872350 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:30.865221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:43089 (local address 127.2.32.62:42743)
0621 05:20:30.865262 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:30.865264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:30.865433 (+ 169us) server_negotiation.cc:407] Connection header received
0621 05:20:30.865528 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:30.865529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:30.865539 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:30.865569 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:30.866085 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:30.866536 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:30.867668 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:30.867932 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:30.867993 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:30.869067 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:30.869069 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:30.869069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:30.869075 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:30.870578 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:30.870956 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:30.870958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:30.870958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:30.871059 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:30.871483 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:30.871484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:30.871485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:30.871645 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:30.871715 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:30.872019 (+ 304us) server_negotiation.cc:299] Negotiation successful
0621 05:20:30.872242 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
I20260621 05:20:30.890272 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:42743
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:30.974713 3643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:30.974911 3643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:30.974936 3643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:30.976209 3643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:30.976254 3643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:30.976275 3643 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:20:30.976293 3643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:20:30.977658 3643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.32.62:42743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:30.977869 3643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:30.978106 3643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:30.983522 3649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:30.983552 3651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:30.983565 3648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:30.983764 3643 server_base.cc:1034] running on GCE node
I20260621 05:20:30.984050 3643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:30.984304 3643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:30.985459 3643 hybrid_clock.cc:648] HybridClock initialized: now 1782019230985449 us; error 38 us; skew 500 ppm
Jun 21 05:20:30 dist-test-slave-b357 krb5kdc[3246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019230, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:30.988761 3643 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:20:30.989449 3643 webserver.cc:460] Webserver started at http://127.2.32.3:45867/ using document root <none> and password file <none>
I20260621 05:20:30.989755 3643 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:30.989837 3643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:30.990006 3643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:30.991309 3643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "60ef4cd12ace492e9b4fc35fed37d033"
format_stamp: "Formatted at 2026-06-21 05:20:30 on dist-test-slave-b357"
server_key: "b7bf1613c9c4860310c6e15caf7e9a83"
server_key_iv: "6ae4f1f116c508fea84dab91f0282dcf"
server_key_version: "encryptionkey@0"
I20260621 05:20:30.991712 3643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "60ef4cd12ace492e9b4fc35fed37d033"
format_stamp: "Formatted at 2026-06-21 05:20:30 on dist-test-slave-b357"
server_key: "b7bf1613c9c4860310c6e15caf7e9a83"
server_key_iv: "6ae4f1f116c508fea84dab91f0282dcf"
server_key_version: "encryptionkey@0"
I20260621 05:20:30.993500 3643 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260621 05:20:30.994398 3658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:30.994608 3643 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:30.994717 3643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "60ef4cd12ace492e9b4fc35fed37d033"
format_stamp: "Formatted at 2026-06-21 05:20:30 on dist-test-slave-b357"
server_key: "b7bf1613c9c4860310c6e15caf7e9a83"
server_key_iv: "6ae4f1f116c508fea84dab91f0282dcf"
server_key_version: "encryptionkey@0"
I20260621 05:20:30.994822 3643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:31.007510 3643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:31.009821 3643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:31.010092 3643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:31.010604 3643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:31.010663 3643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:31.010696 3643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:31.010728 3643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:31.018392 3643 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:45657
Jun 21 05:20:31 dist-test-slave-b357 krb5kdc[3246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019230, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:31.018440 3773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:45657 every 8 connection(s)
I20260621 05:20:31.018813 3643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:20:31.022365 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.011505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:42743 (local address 127.2.32.3:35489)
0621 05:20:31.011783 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:31.011835 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:31.012674 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:31.013022 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:31.013026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:31.013399 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:31.013695 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.013701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.014732 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.014733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.016515 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.016518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.016946 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.016947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.017070 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.017624 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:31.017632 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:31.019273 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:31.021012 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:31.021018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:31.021020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:31.021247 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:31.021531 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:31.021532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:31.021532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:31.021625 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:31.021919 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:31.021923 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:31.022029 (+ 106us) client_negotiation.cc:769] Sending connection context
0621 05:20:31.022111 (+ 82us) client_negotiation.cc:240] Negotiation successful
0621 05:20:31.022222 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":162,"threads_started":1}
I20260621 05:20:31.022387 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.011557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:35489 (local address 127.2.32.62:42743)
0621 05:20:31.011645 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:31.011649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:31.011977 (+ 328us) server_negotiation.cc:407] Connection header received
0621 05:20:31.012818 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:31.012819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:31.012830 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:31.012916 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:31.013798 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.014631 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.016631 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.016821 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.016934 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.019366 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:31.019367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:31.019367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:31.019378 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:31.020893 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:31.021362 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:31.021363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:31.021364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:31.021441 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:31.021722 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:31.021723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:31.021723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:31.021851 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:31.021920 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:31.022191 (+ 271us) server_negotiation.cc:299] Negotiation successful
0621 05:20:31.022305 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 05:20:31.023295 3774 heartbeater.cc:347] Connected to a master server at 127.2.32.62:42743
I20260621 05:20:31.023411 3774 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:31.023641 3774 heartbeater.cc:511] Master 127.2.32.62:42743 requested a full tablet report, sending...
I20260621 05:20:31.024145 3290 ts_manager.cc:194] Registered new tserver with Master: 60ef4cd12ace492e9b4fc35fed37d033 (127.2.32.3:45657)
I20260621 05:20:31.024632 3290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:35489
I20260621 05:20:31.025976 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 3643
I20260621 05:20:31.026134 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:20:31.026376 2176 external_mini_cluster.cc:1428] Setting key 9d953c39e3eeac293aeccb768554b0a9
I20260621 05:20:31.027259 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:20:31.031549 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.024576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:47773 (local address 127.2.32.62:42743)
0621 05:20:31.024635 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:31.024637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:31.024770 (+ 133us) server_negotiation.cc:407] Connection header received
0621 05:20:31.024842 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:31.024843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:31.024852 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:31.024898 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:31.025230 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.025695 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.026756 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.027018 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.027099 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.028463 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:31.028465 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:31.028465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:31.028473 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:31.030168 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:31.030671 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:31.030672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:31.030673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:31.030728 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:31.030991 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:31.030992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:31.030993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:31.031100 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:31.031220 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:31.031378 (+ 158us) server_negotiation.cc:299] Negotiation successful
0621 05:20:31.031456 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260621 05:20:31.031576 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.024506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:42743 (local address 127.2.32.3:47773)
0621 05:20:31.024709 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:31.024721 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:31.024784 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:31.024900 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:31.024901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:31.025087 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:31.025141 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.025142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.025789 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.025790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.026666 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.026669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.027099 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.027100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.027205 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.027854 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:31.027860 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:31.028337 (+ 477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:31.030282 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:31.030284 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:31.030285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:31.030557 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:31.030824 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:31.030825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:31.030825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:31.030907 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:31.031217 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:31.031218 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:31.031265 (+ 47us) client_negotiation.cc:769] Sending connection context
0621 05:20:31.031355 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 05:20:31.031478 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":72,"threads_started":1}
Jun 21 05:20:31 dist-test-slave-b357 krb5kdc[3246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019231, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 05:20:31 dist-test-slave-b357 krb5kdc[3246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019231, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:31.053462 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.045453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59932 (local address 127.2.32.62:42743)
0621 05:20:31.045539 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:31.045541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:31.045556 (+ 15us) server_negotiation.cc:407] Connection header received
0621 05:20:31.045698 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:31.045700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:31.045715 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:31.045788 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:31.046389 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.046932 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.047860 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.048085 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.048156 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.050794 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:31.050796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:31.050796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:31.050802 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:31.052216 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:31.052608 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:31.052608 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:31.052609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:31.052658 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:31.052839 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:31.052840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:31.052840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:31.052940 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:31.053016 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:31.053324 (+ 308us) server_negotiation.cc:299] Negotiation successful
0621 05:20:31.053392 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 05:20:31.055763 3290 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:20:31.057682 3290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:20:31.065081 3792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.060621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:45657 (local address 127.0.0.1:35118)
0621 05:20:31.060888 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:31.060929 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:31.061002 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:31.061696 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:31.061697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:31.061700 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:31.061760 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.061761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.062703 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.062704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.063822 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.063824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.064789 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.064790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.064844 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.064853 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 05:20:31.064951 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 05:20:31.065010 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":49,"threads_started":1}
I20260621 05:20:31.065136 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.060777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.2.32.3:45657)
0621 05:20:31.061459 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:31.061463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:31.061473 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:20:31.061515 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:31.061517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:31.061528 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:31.061619 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:31.061819 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.062600 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.063912 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.064637 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.064697 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.064752 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:31.064972 (+ 220us) server_negotiation.cc:299] Negotiation successful
0621 05:20:31.065041 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":113,"threads_started":1}
I20260621 05:20:31.065789 3791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.060700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:41747 (local address 127.0.0.1:51446)
0621 05:20:31.060959 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:31.060968 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:31.061011 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:31.061459 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:31.061464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:31.061468 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:31.061587 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.061589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.063103 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.063104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.064262 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.064265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.065533 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.065534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.065614 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.065619 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:20:31.065682 (+ 63us) client_negotiation.cc:240] Negotiation successful
0621 05:20:31.065727 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":42,"thread_start_us":98,"threads_started":1}
I20260621 05:20:31.065881 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.060964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.2.32.2:41747)
0621 05:20:31.061133 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:31.061137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:31.061146 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:20:31.061268 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:31.061273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:31.061296 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:31.061387 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:31.061702 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.062993 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.064358 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.065457 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.065535 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.065607 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:31.065698 (+ 91us) server_negotiation.cc:299] Negotiation successful
0621 05:20:31.065747 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":71,"threads_started":1}
I20260621 05:20:31.066094 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.060709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:37679 (local address 127.0.0.1:37900)
0621 05:20:31.061190 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:31.061201 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:31.061334 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:31.061746 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:31.061747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:31.061749 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:31.061893 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.061895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.063182 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.063183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.064612 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.064615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.065794 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.065795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.065869 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.065875 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:20:31.065929 (+ 54us) client_negotiation.cc:240] Negotiation successful
0621 05:20:31.065990 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":105,"threads_started":1}
I20260621 05:20:31.065960 3708 tablet_service.cc:1467] Processing CreateTablet for tablet 3042e01f6b4c4c1dbea1ee3dafd43b30 (DEFAULT_TABLE table=test-table [id=8f1004f0b9364864bc560fd5bd9a630e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:31.066325 3708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3042e01f6b4c4c1dbea1ee3dafd43b30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:31.066438 3795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.060953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.2.32.1:37679)
0621 05:20:31.061119 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:31.061123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:31.061433 (+ 310us) server_negotiation.cc:407] Connection header received
0621 05:20:31.061497 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:31.061501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:31.061521 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:31.061638 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:31.061978 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.063062 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.064711 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.065406 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.065694 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.065762 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:31.066266 (+ 504us) server_negotiation.cc:299] Negotiation successful
0621 05:20:31.066324 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260621 05:20:31.066478 3568 tablet_service.cc:1467] Processing CreateTablet for tablet 3042e01f6b4c4c1dbea1ee3dafd43b30 (DEFAULT_TABLE table=test-table [id=8f1004f0b9364864bc560fd5bd9a630e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:31.066768 3568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3042e01f6b4c4c1dbea1ee3dafd43b30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:31.067075 3428 tablet_service.cc:1467] Processing CreateTablet for tablet 3042e01f6b4c4c1dbea1ee3dafd43b30 (DEFAULT_TABLE table=test-table [id=8f1004f0b9364864bc560fd5bd9a630e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:31.067591 3428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3042e01f6b4c4c1dbea1ee3dafd43b30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:31.069751 3797 tablet_bootstrap.cc:492] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033: Bootstrap starting.
I20260621 05:20:31.070240 3798 tablet_bootstrap.cc:492] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a: Bootstrap starting.
I20260621 05:20:31.070798 3798 tablet_bootstrap.cc:654] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:31.071107 3797 tablet_bootstrap.cc:654] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:31.071190 3798 log.cc:824] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:31.071527 3799 tablet_bootstrap.cc:492] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16: Bootstrap starting.
I20260621 05:20:31.071674 3797 log.cc:824] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:31.072158 3799 tablet_bootstrap.cc:654] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:31.072520 3799 log.cc:824] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:31.072645 3797 tablet_bootstrap.cc:492] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033: No bootstrap required, opened a new log
I20260621 05:20:31.072714 3797 ts_tablet_manager.cc:1397] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20260621 05:20:31.072844 3798 tablet_bootstrap.cc:492] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a: No bootstrap required, opened a new log
I20260621 05:20:31.072918 3798 ts_tablet_manager.cc:1397] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260621 05:20:31.073304 3799 tablet_bootstrap.cc:492] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16: No bootstrap required, opened a new log
I20260621 05:20:31.073359 3799 ts_tablet_manager.cc:1397] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:20:31.074232 3798 raft_consensus.cc:348] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } } peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } }
I20260621 05:20:31.074421 3798 raft_consensus.cc:374] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:31.074446 3798 raft_consensus.cc:729] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 347c0427a88e45c5b1fad396bbe0f26a, State: Initialized, Role: FOLLOWER
I20260621 05:20:31.074558 3798 consensus_queue.cc:260] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } } peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } }
I20260621 05:20:31.074661 3797 raft_consensus.cc:348] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } } peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } }
I20260621 05:20:31.074813 3798 ts_tablet_manager.cc:1428] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 05:20:31.074808 3797 raft_consensus.cc:374] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:31.074843 3797 raft_consensus.cc:729] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ef4cd12ace492e9b4fc35fed37d033, State: Initialized, Role: FOLLOWER
I20260621 05:20:31.074928 3634 heartbeater.cc:503] Master 127.2.32.62:42743 was elected leader, sending a full tablet report...
I20260621 05:20:31.075305 3797 consensus_queue.cc:260] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } } peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } }
I20260621 05:20:31.075609 3797 ts_tablet_manager.cc:1428] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 05:20:31.075639 3774 heartbeater.cc:503] Master 127.2.32.62:42743 was elected leader, sending a full tablet report...
I20260621 05:20:31.076157 3799 raft_consensus.cc:348] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } } peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } }
I20260621 05:20:31.076313 3799 raft_consensus.cc:374] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:31.076359 3799 raft_consensus.cc:729] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 161d3b6a40d44954a10ffed41a6f0a16, State: Initialized, Role: FOLLOWER
I20260621 05:20:31.076478 3799 consensus_queue.cc:260] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } } peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } }
I20260621 05:20:31.076762 3494 heartbeater.cc:503] Master 127.2.32.62:42743 was elected leader, sending a full tablet report...
I20260621 05:20:31.077132 3799 ts_tablet_manager.cc:1428] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260621 05:20:31.086004 3803 raft_consensus.cc:482] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:20:31.086154 3803 raft_consensus.cc:504] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } } peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } }
I20260621 05:20:31.086537 3803 leader_election.cc:288] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ef4cd12ace492e9b4fc35fed37d033 (127.2.32.3:45657), 161d3b6a40d44954a10ffed41a6f0a16 (127.2.32.1:37679)
I20260621 05:20:31.090358 3527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.086806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:37679 (local address 127.2.32.2:49765)
0621 05:20:31.086883 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:31.086897 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:31.086960 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:31.087213 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:31.087215 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:31.087217 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:31.087340 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.087342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.088154 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.088155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.089405 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.089407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.090080 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.090081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.090178 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.090184 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:20:31.090245 (+ 61us) client_negotiation.cc:240] Negotiation successful
0621 05:20:31.090299 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":34}
I20260621 05:20:31.090453 3795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.086846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:49765 (local address 127.2.32.1:37679)
0621 05:20:31.086898 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:31.086900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:31.087008 (+ 108us) server_negotiation.cc:407] Connection header received
0621 05:20:31.087042 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:31.087043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:31.087058 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:31.087165 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:31.087442 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.088048 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.089492 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.089975 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.090050 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.090063 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:31.090289 (+ 226us) server_negotiation.cc:299] Negotiation successful
0621 05:20:31.090374 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260621 05:20:31.090595 3806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.086816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:45657 (local address 127.2.32.2:33667)
0621 05:20:31.087079 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:31.087089 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:31.087185 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:31.087416 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:31.087418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:31.087421 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:31.087529 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.087531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.088387 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.088388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.089562 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.089564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.090250 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.090251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.090349 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.090353 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:20:31.090423 (+ 70us) client_negotiation.cc:240] Negotiation successful
0621 05:20:31.090472 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":18,"thread_start_us":111,"threads_started":1}
I20260621 05:20:31.090677 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.086918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:33667 (local address 127.2.32.3:45657)
0621 05:20:31.086985 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:31.086988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:31.087197 (+ 209us) server_negotiation.cc:407] Connection header received
0621 05:20:31.087251 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:31.087252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:31.087265 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:31.087342 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:31.087669 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.088276 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.089689 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.090111 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.090191 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.090202 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:31.090581 (+ 379us) server_negotiation.cc:299] Negotiation successful
0621 05:20:31.090619 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 05:20:31.090716 3448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3042e01f6b4c4c1dbea1ee3dafd43b30" candidate_uuid: "347c0427a88e45c5b1fad396bbe0f26a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "161d3b6a40d44954a10ffed41a6f0a16" is_pre_election: true
I20260621 05:20:31.090870 3448 raft_consensus.cc:2454] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 347c0427a88e45c5b1fad396bbe0f26a in term 0.
I20260621 05:20:31.090955 3728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3042e01f6b4c4c1dbea1ee3dafd43b30" candidate_uuid: "347c0427a88e45c5b1fad396bbe0f26a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ef4cd12ace492e9b4fc35fed37d033" is_pre_election: true
I20260621 05:20:31.091073 3728 raft_consensus.cc:2454] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 347c0427a88e45c5b1fad396bbe0f26a in term 0.
I20260621 05:20:31.091091 3522 leader_election.cc:302] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 161d3b6a40d44954a10ffed41a6f0a16, 347c0427a88e45c5b1fad396bbe0f26a; no voters:
I20260621 05:20:31.091231 3803 raft_consensus.cc:2781] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:20:31.091292 3803 raft_consensus.cc:482] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:20:31.091310 3803 raft_consensus.cc:3037] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:31.092231 3803 raft_consensus.cc:504] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } } peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } }
I20260621 05:20:31.092376 3803 leader_election.cc:288] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [CANDIDATE]: Term 1 election: Requested vote from peers 60ef4cd12ace492e9b4fc35fed37d033 (127.2.32.3:45657), 161d3b6a40d44954a10ffed41a6f0a16 (127.2.32.1:37679)
I20260621 05:20:31.092526 3448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3042e01f6b4c4c1dbea1ee3dafd43b30" candidate_uuid: "347c0427a88e45c5b1fad396bbe0f26a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "161d3b6a40d44954a10ffed41a6f0a16"
I20260621 05:20:31.092610 3448 raft_consensus.cc:3037] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:31.092636 3728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3042e01f6b4c4c1dbea1ee3dafd43b30" candidate_uuid: "347c0427a88e45c5b1fad396bbe0f26a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ef4cd12ace492e9b4fc35fed37d033"
I20260621 05:20:31.092694 3728 raft_consensus.cc:3037] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:31.093495 3448 raft_consensus.cc:2454] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 347c0427a88e45c5b1fad396bbe0f26a in term 1.
I20260621 05:20:31.093504 3728 raft_consensus.cc:2454] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 347c0427a88e45c5b1fad396bbe0f26a in term 1.
I20260621 05:20:31.093648 3522 leader_election.cc:302] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 161d3b6a40d44954a10ffed41a6f0a16, 347c0427a88e45c5b1fad396bbe0f26a; no voters:
I20260621 05:20:31.093737 3803 raft_consensus.cc:2781] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:31.093858 3803 raft_consensus.cc:686] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [term 1 LEADER]: Becoming Leader. State: Replica: 347c0427a88e45c5b1fad396bbe0f26a, State: Running, Role: LEADER
I20260621 05:20:31.093953 3803 consensus_queue.cc:237] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } } peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } }
I20260621 05:20:31.094771 3290 catalog_manager.cc:5495] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a reported cstate change: term changed from 0 to 1, leader changed from <none> to 347c0427a88e45c5b1fad396bbe0f26a (127.2.32.2). New cstate: current_term: 1 leader_uuid: "347c0427a88e45c5b1fad396bbe0f26a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } health_report { overall_health: HEALTHY } } }
I20260621 05:20:31.102970 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.100426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51458 (local address 127.2.32.2:41747)
0621 05:20:31.100502 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:31.100504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:31.100512 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:20:31.100538 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:31.100539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:31.100557 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:31.100635 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:31.100778 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.101238 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.102100 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.102304 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.102378 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.102605 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:31.102697 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:31.102789 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:31.102851 (+ 62us) server_negotiation.cc:299] Negotiation successful
0621 05:20:31.102866 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 05:20:31.104691 3448 raft_consensus.cc:1261] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16 [term 1 FOLLOWER]: Refusing update from remote peer 347c0427a88e45c5b1fad396bbe0f26a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:31.104660 3728 raft_consensus.cc:1261] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033 [term 1 FOLLOWER]: Refusing update from remote peer 347c0427a88e45c5b1fad396bbe0f26a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:31.105008 3803 consensus_queue.cc:1035] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:31.105126 3807 consensus_queue.cc:1035] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [LEADER]: Connected to new peer: Peer: permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:31.107432 3810 mvcc.cc:205] Tried to move back new op lower bound from 7299150770601922560 to 7299150770561323008. Current Snapshot: MvccSnapshot[applied={T|T < 7299150770601922560}]
I20260621 05:20:31.110999 3291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:20:31.111105 3291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:20:31.112139 3291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf: Sending DeleteTablet for 3 replicas of tablet 3042e01f6b4c4c1dbea1ee3dafd43b30
I20260621 05:20:31.112452 3428 tablet_service.cc:1514] Processing DeleteTablet for tablet 3042e01f6b4c4c1dbea1ee3dafd43b30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:31 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:37900
I20260621 05:20:31.112478 3708 tablet_service.cc:1514] Processing DeleteTablet for tablet 3042e01f6b4c4c1dbea1ee3dafd43b30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:31 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:35118
I20260621 05:20:31.112661 3568 tablet_service.cc:1514] Processing DeleteTablet for tablet 3042e01f6b4c4c1dbea1ee3dafd43b30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:31 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:51446
I20260621 05:20:31.112691 3821 tablet_replica.cc:330] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16: stopping tablet replica
I20260621 05:20:31.112785 3821 raft_consensus.cc:2229] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:20:31.112859 3822 tablet_replica.cc:330] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a: stopping tablet replica
I20260621 05:20:31.112883 3821 raft_consensus.cc:2258] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:31.112941 3822 raft_consensus.cc:2229] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:20:31.113046 3822 raft_consensus.cc:2258] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:31.113216 3291 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:20:31.113366 3291 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:20:31.113405 3821 ts_tablet_manager.cc:1905] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:31.113984 3822 ts_tablet_manager.cc:1905] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:31.114773 3708 tablet_service.cc:1467] Processing CreateTablet for tablet 57b8dd93c6344dfe8d40fb3d908d74ad (DEFAULT_TABLE table=test-table [id=efdd3be218e64e7296bd0e2b0f5bb105]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:31.114893 3708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b8dd93c6344dfe8d40fb3d908d74ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:31.114907 3428 tablet_service.cc:1467] Processing CreateTablet for tablet 57b8dd93c6344dfe8d40fb3d908d74ad (DEFAULT_TABLE table=test-table [id=efdd3be218e64e7296bd0e2b0f5bb105]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:31.115003 3428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b8dd93c6344dfe8d40fb3d908d74ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:31.115373 3821 ts_tablet_manager.cc:1918] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:20:31.115442 3821 log.cc:1196] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-0/wal/wals/3042e01f6b4c4c1dbea1ee3dafd43b30
I20260621 05:20:31.115689 3821 ts_tablet_manager.cc:1939] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 161d3b6a40d44954a10ffed41a6f0a16: Deleting consensus metadata
I20260621 05:20:31.116053 3278 catalog_manager.cc:4841] TS 161d3b6a40d44954a10ffed41a6f0a16 (127.2.32.1:37679): tablet 3042e01f6b4c4c1dbea1ee3dafd43b30 (table test-table [id=8f1004f0b9364864bc560fd5bd9a630e]) successfully deleted
I20260621 05:20:31.116204 3799 tablet_bootstrap.cc:492] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16: Bootstrap starting.
I20260621 05:20:31.116250 3797 tablet_bootstrap.cc:492] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033: Bootstrap starting.
I20260621 05:20:31.116302 3820 tablet_replica.cc:330] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033: stopping tablet replica
I20260621 05:20:31.116403 3820 raft_consensus.cc:2229] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:20:31.116495 3820 raft_consensus.cc:2258] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:31.116623 3799 tablet_bootstrap.cc:654] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:31.116791 3797 tablet_bootstrap.cc:654] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:31.116890 3820 ts_tablet_manager.cc:1905] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:31.117213 3568 tablet_service.cc:1467] Processing CreateTablet for tablet 57b8dd93c6344dfe8d40fb3d908d74ad (DEFAULT_TABLE table=test-table [id=efdd3be218e64e7296bd0e2b0f5bb105]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:31.117319 3568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b8dd93c6344dfe8d40fb3d908d74ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:31.118258 3798 tablet_bootstrap.cc:492] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a: Bootstrap starting.
I20260621 05:20:31.118278 3820 ts_tablet_manager.cc:1918] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:20:31.118335 3820 log.cc:1196] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/3042e01f6b4c4c1dbea1ee3dafd43b30
I20260621 05:20:31.118588 3820 ts_tablet_manager.cc:1939] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 60ef4cd12ace492e9b4fc35fed37d033: Deleting consensus metadata
I20260621 05:20:31.118919 3798 tablet_bootstrap.cc:654] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:31.119035 3275 catalog_manager.cc:4841] TS 60ef4cd12ace492e9b4fc35fed37d033 (127.2.32.3:45657): tablet 3042e01f6b4c4c1dbea1ee3dafd43b30 (table test-table [id=8f1004f0b9364864bc560fd5bd9a630e]) successfully deleted
I20260621 05:20:31.119644 3797 tablet_bootstrap.cc:492] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033: No bootstrap required, opened a new log
I20260621 05:20:31.119696 3797 ts_tablet_manager.cc:1397] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260621 05:20:31.119822 3797 raft_consensus.cc:348] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } }
I20260621 05:20:31.119899 3797 raft_consensus.cc:374] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:31.119921 3797 raft_consensus.cc:729] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ef4cd12ace492e9b4fc35fed37d033, State: Initialized, Role: FOLLOWER
I20260621 05:20:31.119961 3797 consensus_queue.cc:260] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } }
I20260621 05:20:31.120040 3797 ts_tablet_manager.cc:1428] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:31.120309 3799 tablet_bootstrap.cc:492] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16: No bootstrap required, opened a new log
I20260621 05:20:31.120353 3799 ts_tablet_manager.cc:1397] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260621 05:20:31.120497 3799 raft_consensus.cc:348] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } }
I20260621 05:20:31.120563 3799 raft_consensus.cc:374] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:31.120589 3799 raft_consensus.cc:729] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 161d3b6a40d44954a10ffed41a6f0a16, State: Initialized, Role: FOLLOWER
I20260621 05:20:31.120626 3799 consensus_queue.cc:260] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } }
I20260621 05:20:31.120718 3799 ts_tablet_manager.cc:1428] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:31.120990 3822 ts_tablet_manager.cc:1918] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:20:31.121099 3822 log.cc:1196] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/3042e01f6b4c4c1dbea1ee3dafd43b30
I20260621 05:20:31.121549 3822 ts_tablet_manager.cc:1939] T 3042e01f6b4c4c1dbea1ee3dafd43b30 P 347c0427a88e45c5b1fad396bbe0f26a: Deleting consensus metadata
I20260621 05:20:31.121912 3277 catalog_manager.cc:4841] TS 347c0427a88e45c5b1fad396bbe0f26a (127.2.32.2:41747): tablet 3042e01f6b4c4c1dbea1ee3dafd43b30 (table test-table [id=8f1004f0b9364864bc560fd5bd9a630e]) successfully deleted
I20260621 05:20:31.122342 3798 tablet_bootstrap.cc:492] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a: No bootstrap required, opened a new log
I20260621 05:20:31.122393 3798 ts_tablet_manager.cc:1397] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260621 05:20:31.122557 3798 raft_consensus.cc:348] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } }
I20260621 05:20:31.122623 3798 raft_consensus.cc:374] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:31.122648 3798 raft_consensus.cc:729] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 347c0427a88e45c5b1fad396bbe0f26a, State: Initialized, Role: FOLLOWER
I20260621 05:20:31.122678 3798 consensus_queue.cc:260] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } }
I20260621 05:20:31.122762 3798 ts_tablet_manager.cc:1428] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260621 05:20:31.198395 3495 tablet.cc:2369] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:20:31.269487 3775 tablet.cc:2369] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:20:31.321076 3804 raft_consensus.cc:482] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:20:31.321193 3804 raft_consensus.cc:504] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } }
I20260621 05:20:31.321651 3804 leader_election.cc:288] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 161d3b6a40d44954a10ffed41a6f0a16 (127.2.32.1:37679), 347c0427a88e45c5b1fad396bbe0f26a (127.2.32.2:41747)
I20260621 05:20:31.325141 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.321889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:37679 (local address 127.2.32.3:60287)
0621 05:20:31.321966 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:31.321980 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:31.322046 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:31.322191 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:31.322192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:31.322194 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:31.322269 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.322271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.322992 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.322994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.324145 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.324154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.324854 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.324855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.324950 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.324954 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:20:31.325020 (+ 66us) client_negotiation.cc:240] Negotiation successful
0621 05:20:31.325069 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 05:20:31.325350 3795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.321983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:60287 (local address 127.2.32.1:37679)
0621 05:20:31.322042 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:31.322045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:31.322062 (+ 17us) server_negotiation.cc:407] Connection header received
0621 05:20:31.322105 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:31.322107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:31.322118 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:31.322176 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:31.322352 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.322887 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.324246 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.324759 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.324817 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.324826 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:31.325123 (+ 297us) server_negotiation.cc:299] Negotiation successful
0621 05:20:31.325192 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:20:31.325591 3826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.321897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:41747 (local address 127.2.32.3:50535)
0621 05:20:31.322059 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:31.322070 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:31.322153 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:31.322300 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:31.322301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:31.322302 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:31.322420 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.322421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.323528 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.323529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.324531 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.324534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.325332 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.325333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.325393 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.325396 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:20:31.325467 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 05:20:31.325504 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":16,"thread_start_us":76,"threads_started":1}
I20260621 05:20:31.325688 3448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b8dd93c6344dfe8d40fb3d908d74ad" candidate_uuid: "60ef4cd12ace492e9b4fc35fed37d033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "161d3b6a40d44954a10ffed41a6f0a16" is_pre_election: true
I20260621 05:20:31.325774 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.321990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:50535 (local address 127.2.32.2:41747)
0621 05:20:31.322036 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:31.322039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:31.322176 (+ 137us) server_negotiation.cc:407] Connection header received
0621 05:20:31.322219 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:31.322220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:31.322229 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:31.322285 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:31.322659 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.323381 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.324644 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.325199 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.325268 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.325278 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:31.325556 (+ 278us) server_negotiation.cc:299] Negotiation successful
0621 05:20:31.325639 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:20:31.325789 3448 raft_consensus.cc:2454] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ef4cd12ace492e9b4fc35fed37d033 in term 0.
I20260621 05:20:31.325996 3662 leader_election.cc:302] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 161d3b6a40d44954a10ffed41a6f0a16, 60ef4cd12ace492e9b4fc35fed37d033; no voters:
I20260621 05:20:31.326151 3804 raft_consensus.cc:2781] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:20:31.326189 3804 raft_consensus.cc:482] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:20:31.326223 3804 raft_consensus.cc:3037] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:31.326061 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b8dd93c6344dfe8d40fb3d908d74ad" candidate_uuid: "60ef4cd12ace492e9b4fc35fed37d033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "347c0427a88e45c5b1fad396bbe0f26a" is_pre_election: true
I20260621 05:20:31.326427 3588 raft_consensus.cc:2454] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ef4cd12ace492e9b4fc35fed37d033 in term 0.
I20260621 05:20:31.327028 3804 raft_consensus.cc:504] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } }
I20260621 05:20:31.327200 3804 leader_election.cc:288] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [CANDIDATE]: Term 1 election: Requested vote from peers 161d3b6a40d44954a10ffed41a6f0a16 (127.2.32.1:37679), 347c0427a88e45c5b1fad396bbe0f26a (127.2.32.2:41747)
I20260621 05:20:31.327335 3448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b8dd93c6344dfe8d40fb3d908d74ad" candidate_uuid: "60ef4cd12ace492e9b4fc35fed37d033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "161d3b6a40d44954a10ffed41a6f0a16"
I20260621 05:20:31.327412 3448 raft_consensus.cc:3037] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:31.327435 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b8dd93c6344dfe8d40fb3d908d74ad" candidate_uuid: "60ef4cd12ace492e9b4fc35fed37d033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "347c0427a88e45c5b1fad396bbe0f26a"
I20260621 05:20:31.327514 3588 raft_consensus.cc:3037] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:31.328039 3448 raft_consensus.cc:2454] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ef4cd12ace492e9b4fc35fed37d033 in term 1.
I20260621 05:20:31.328212 3662 leader_election.cc:302] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 161d3b6a40d44954a10ffed41a6f0a16, 60ef4cd12ace492e9b4fc35fed37d033; no voters:
I20260621 05:20:31.328249 3588 raft_consensus.cc:2454] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ef4cd12ace492e9b4fc35fed37d033 in term 1.
I20260621 05:20:31.328301 3804 raft_consensus.cc:2781] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:31.328452 3804 raft_consensus.cc:686] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [term 1 LEADER]: Becoming Leader. State: Replica: 60ef4cd12ace492e9b4fc35fed37d033, State: Running, Role: LEADER
I20260621 05:20:31.328531 3804 consensus_queue.cc:237] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } }
I20260621 05:20:31.329233 3290 catalog_manager.cc:5495] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ef4cd12ace492e9b4fc35fed37d033 (127.2.32.3). New cstate: current_term: 1 leader_uuid: "60ef4cd12ace492e9b4fc35fed37d033" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } health_report { overall_health: HEALTHY } } }
W20260621 05:20:31.360783 3635 tablet.cc:2369] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 21 05:20:31 dist-test-slave-b357 krb5kdc[3246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019230, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:31.379637 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.372042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59938 (local address 127.2.32.62:42743)
0621 05:20:31.372114 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:31.372116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:31.372263 (+ 147us) server_negotiation.cc:407] Connection header received
0621 05:20:31.372294 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:31.372294 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:31.372304 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:31.372373 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:31.372793 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.373437 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.374154 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.374339 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.374403 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.376995 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:31.376996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:31.376996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:31.377002 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:31.378372 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:31.378937 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:31.378938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:31.378939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:31.378994 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:31.379243 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:31.379243 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:31.379243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:31.379333 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:31.379382 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:31.379507 (+ 125us) server_negotiation.cc:299] Negotiation successful
0621 05:20:31.379574 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 05:20:31.379956 3834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.371988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:42743 (local address 127.0.0.1:59938)
0621 05:20:31.372170 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:31.372181 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:31.372250 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:31.372384 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:31.372384 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:31.372661 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:31.372707 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.372709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.373561 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.373562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.374067 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.374069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.374418 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.374419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.374461 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.375222 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:31.375231 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:31.376914 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:31.378458 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:31.378465 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:31.378466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:31.378825 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:31.379095 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:31.379096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:31.379097 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:31.379181 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:31.379387 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:31.379394 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:31.379434 (+ 40us) client_negotiation.cc:769] Sending connection context
0621 05:20:31.379491 (+ 57us) client_negotiation.cc:240] Negotiation successful
0621 05:20:31.379761 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":99,"thread_start_us":106,"threads_started":1}
I20260621 05:20:31.382342 3290 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:59938:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 05:20:31.382627 3290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:20:31.384323 3568 tablet_service.cc:1467] Processing CreateTablet for tablet e582869bb3104cb0943c01a4117cf2ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b3451c916844b2abdc4f28a5416ebbd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:20:31.384323 3428 tablet_service.cc:1467] Processing CreateTablet for tablet e582869bb3104cb0943c01a4117cf2ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b3451c916844b2abdc4f28a5416ebbd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:20:31.384353 3708 tablet_service.cc:1467] Processing CreateTablet for tablet e582869bb3104cb0943c01a4117cf2ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b3451c916844b2abdc4f28a5416ebbd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:20:31.384472 3568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e582869bb3104cb0943c01a4117cf2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:31.384477 3428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e582869bb3104cb0943c01a4117cf2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:31.384474 3708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e582869bb3104cb0943c01a4117cf2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:31.385985 3798 tablet_bootstrap.cc:492] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a: Bootstrap starting.
I20260621 05:20:31.386056 3799 tablet_bootstrap.cc:492] T e582869bb3104cb0943c01a4117cf2ea P 161d3b6a40d44954a10ffed41a6f0a16: Bootstrap starting.
I20260621 05:20:31.386117 3797 tablet_bootstrap.cc:492] T e582869bb3104cb0943c01a4117cf2ea P 60ef4cd12ace492e9b4fc35fed37d033: Bootstrap starting.
I20260621 05:20:31.386374 3798 tablet_bootstrap.cc:654] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:31.386768 3799 tablet_bootstrap.cc:654] T e582869bb3104cb0943c01a4117cf2ea P 161d3b6a40d44954a10ffed41a6f0a16: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:31.386768 3797 tablet_bootstrap.cc:654] T e582869bb3104cb0943c01a4117cf2ea P 60ef4cd12ace492e9b4fc35fed37d033: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:31.387163 3798 tablet_bootstrap.cc:492] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a: No bootstrap required, opened a new log
I20260621 05:20:31.387219 3798 ts_tablet_manager.cc:1397] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:31.387346 3798 raft_consensus.cc:348] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } }
I20260621 05:20:31.387439 3798 raft_consensus.cc:374] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:31.387465 3798 raft_consensus.cc:729] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 347c0427a88e45c5b1fad396bbe0f26a, State: Initialized, Role: FOLLOWER
I20260621 05:20:31.387523 3798 consensus_queue.cc:260] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } }
I20260621 05:20:31.387655 3798 ts_tablet_manager.cc:1428] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:31.387665 3803 tablet_replica.cc:439] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } } }
I20260621 05:20:31.387784 3803 tablet_replica.cc:442] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:20:31.387997 3799 tablet_bootstrap.cc:492] T e582869bb3104cb0943c01a4117cf2ea P 161d3b6a40d44954a10ffed41a6f0a16: No bootstrap required, opened a new log
I20260621 05:20:31.388052 3799 ts_tablet_manager.cc:1397] T e582869bb3104cb0943c01a4117cf2ea P 161d3b6a40d44954a10ffed41a6f0a16: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:20:31.388252 3799 raft_consensus.cc:348] T e582869bb3104cb0943c01a4117cf2ea P 161d3b6a40d44954a10ffed41a6f0a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } }
I20260621 05:20:31.388341 3799 raft_consensus.cc:374] T e582869bb3104cb0943c01a4117cf2ea P 161d3b6a40d44954a10ffed41a6f0a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:31.388372 3799 raft_consensus.cc:729] T e582869bb3104cb0943c01a4117cf2ea P 161d3b6a40d44954a10ffed41a6f0a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 161d3b6a40d44954a10ffed41a6f0a16, State: Initialized, Role: FOLLOWER
I20260621 05:20:31.388404 3797 tablet_bootstrap.cc:492] T e582869bb3104cb0943c01a4117cf2ea P 60ef4cd12ace492e9b4fc35fed37d033: No bootstrap required, opened a new log
I20260621 05:20:31.388458 3797 ts_tablet_manager.cc:1397] T e582869bb3104cb0943c01a4117cf2ea P 60ef4cd12ace492e9b4fc35fed37d033: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:20:31.388443 3799 consensus_queue.cc:260] T e582869bb3104cb0943c01a4117cf2ea P 161d3b6a40d44954a10ffed41a6f0a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } }
I20260621 05:20:31.388557 3799 ts_tablet_manager.cc:1428] T e582869bb3104cb0943c01a4117cf2ea P 161d3b6a40d44954a10ffed41a6f0a16: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:31.388641 3797 raft_consensus.cc:348] T e582869bb3104cb0943c01a4117cf2ea P 60ef4cd12ace492e9b4fc35fed37d033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } }
I20260621 05:20:31.388703 3797 raft_consensus.cc:374] T e582869bb3104cb0943c01a4117cf2ea P 60ef4cd12ace492e9b4fc35fed37d033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:31.388625 3805 tablet_replica.cc:439] T e582869bb3104cb0943c01a4117cf2ea P 161d3b6a40d44954a10ffed41a6f0a16: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } } }
I20260621 05:20:31.388736 3797 raft_consensus.cc:729] T e582869bb3104cb0943c01a4117cf2ea P 60ef4cd12ace492e9b4fc35fed37d033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ef4cd12ace492e9b4fc35fed37d033, State: Initialized, Role: FOLLOWER
I20260621 05:20:31.388741 3805 tablet_replica.cc:442] T e582869bb3104cb0943c01a4117cf2ea P 161d3b6a40d44954a10ffed41a6f0a16: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:20:31.388782 3797 consensus_queue.cc:260] T e582869bb3104cb0943c01a4117cf2ea P 60ef4cd12ace492e9b4fc35fed37d033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } }
I20260621 05:20:31.388871 3797 ts_tablet_manager.cc:1428] T e582869bb3104cb0943c01a4117cf2ea P 60ef4cd12ace492e9b4fc35fed37d033: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:31.388989 3804 tablet_replica.cc:439] T e582869bb3104cb0943c01a4117cf2ea P 60ef4cd12ace492e9b4fc35fed37d033: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } } }
I20260621 05:20:31.389096 3804 tablet_replica.cc:442] T e582869bb3104cb0943c01a4117cf2ea P 60ef4cd12ace492e9b4fc35fed37d033: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:20:31.399343 3803 raft_consensus.cc:482] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:20:31.399418 3803 raft_consensus.cc:504] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } }
I20260621 05:20:31.399564 3803 leader_election.cc:288] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 161d3b6a40d44954a10ffed41a6f0a16 (127.2.32.1:37679), 60ef4cd12ace492e9b4fc35fed37d033 (127.2.32.3:45657)
I20260621 05:20:31.399791 3448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e582869bb3104cb0943c01a4117cf2ea" candidate_uuid: "347c0427a88e45c5b1fad396bbe0f26a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "161d3b6a40d44954a10ffed41a6f0a16" is_pre_election: true
I20260621 05:20:31.399780 3728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e582869bb3104cb0943c01a4117cf2ea" candidate_uuid: "347c0427a88e45c5b1fad396bbe0f26a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ef4cd12ace492e9b4fc35fed37d033" is_pre_election: true
I20260621 05:20:31.399880 3448 raft_consensus.cc:2454] T e582869bb3104cb0943c01a4117cf2ea P 161d3b6a40d44954a10ffed41a6f0a16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 347c0427a88e45c5b1fad396bbe0f26a in term 0.
I20260621 05:20:31.399887 3728 raft_consensus.cc:2454] T e582869bb3104cb0943c01a4117cf2ea P 60ef4cd12ace492e9b4fc35fed37d033 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 347c0427a88e45c5b1fad396bbe0f26a in term 0.
I20260621 05:20:31.400017 3522 leader_election.cc:302] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 161d3b6a40d44954a10ffed41a6f0a16, 347c0427a88e45c5b1fad396bbe0f26a; no voters:
I20260621 05:20:31.400121 3803 raft_consensus.cc:2781] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:20:31.400173 3803 raft_consensus.cc:482] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:20:31.400211 3803 raft_consensus.cc:3037] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:31.400799 3803 raft_consensus.cc:504] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } }
I20260621 05:20:31.400928 3803 leader_election.cc:288] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a [CANDIDATE]: Term 1 election: Requested vote from peers 161d3b6a40d44954a10ffed41a6f0a16 (127.2.32.1:37679), 60ef4cd12ace492e9b4fc35fed37d033 (127.2.32.3:45657)
I20260621 05:20:31.401041 3448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e582869bb3104cb0943c01a4117cf2ea" candidate_uuid: "347c0427a88e45c5b1fad396bbe0f26a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "161d3b6a40d44954a10ffed41a6f0a16"
I20260621 05:20:31.401113 3448 raft_consensus.cc:3037] T e582869bb3104cb0943c01a4117cf2ea P 161d3b6a40d44954a10ffed41a6f0a16 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:31.401115 3728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e582869bb3104cb0943c01a4117cf2ea" candidate_uuid: "347c0427a88e45c5b1fad396bbe0f26a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ef4cd12ace492e9b4fc35fed37d033"
I20260621 05:20:31.401201 3728 raft_consensus.cc:3037] T e582869bb3104cb0943c01a4117cf2ea P 60ef4cd12ace492e9b4fc35fed37d033 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:31.401661 3448 raft_consensus.cc:2454] T e582869bb3104cb0943c01a4117cf2ea P 161d3b6a40d44954a10ffed41a6f0a16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 347c0427a88e45c5b1fad396bbe0f26a in term 1.
I20260621 05:20:31.401924 3522 leader_election.cc:302] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 161d3b6a40d44954a10ffed41a6f0a16, 347c0427a88e45c5b1fad396bbe0f26a; no voters:
I20260621 05:20:31.402005 3803 raft_consensus.cc:2781] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:31.402056 3803 raft_consensus.cc:686] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a [term 1 LEADER]: Becoming Leader. State: Replica: 347c0427a88e45c5b1fad396bbe0f26a, State: Running, Role: LEADER
I20260621 05:20:31.402112 3803 consensus_queue.cc:237] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } }
I20260621 05:20:31.402371 3803 tablet_replica.cc:439] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a: TxnStatusTablet state changed. Reason: New leader 347c0427a88e45c5b1fad396bbe0f26a. Latest consensus state: current_term: 1 leader_uuid: "347c0427a88e45c5b1fad396bbe0f26a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } } }
I20260621 05:20:31.402441 3803 tablet_replica.cc:442] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:20:31.402472 3728 raft_consensus.cc:2454] T e582869bb3104cb0943c01a4117cf2ea P 60ef4cd12ace492e9b4fc35fed37d033 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 347c0427a88e45c5b1fad396bbe0f26a in term 1.
I20260621 05:20:31.402634 3839 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 05:20:31.402741 3839 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 05:20:31.402688 3291 catalog_manager.cc:5495] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a reported cstate change: term changed from 0 to 1, leader changed from <none> to 347c0427a88e45c5b1fad396bbe0f26a (127.2.32.2). New cstate: current_term: 1 leader_uuid: "347c0427a88e45c5b1fad396bbe0f26a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } health_report { overall_health: UNKNOWN } } }
I20260621 05:20:31.413029 3834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.410056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:41747 (local address 127.0.0.1:51460)
0621 05:20:31.410167 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:31.410184 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:31.410257 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:31.410482 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:31.410484 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:31.410485 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:20:31.410551 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.410555 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.411517 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.411519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.412106 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.412108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.412578 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.412579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.412634 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.412649 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:20:31.412848 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:20:31.412859 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 05:20:31.412914 (+ 55us) client_negotiation.cc:240] Negotiation successful
0621 05:20:31.412967 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260621 05:20:31.413100 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.410172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.2.32.2:41747)
0621 05:20:31.410251 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:31.410254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:31.410268 (+ 14us) server_negotiation.cc:407] Connection header received
0621 05:20:31.410325 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:31.410326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:31.410341 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:31.410421 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:31.410645 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.411425 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.412189 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.412492 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.412561 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.412735 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:31.412768 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:31.412821 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:31.413005 (+ 184us) server_negotiation.cc:299] Negotiation successful
0621 05:20:31.413030 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 05:20:31.414162 3448 raft_consensus.cc:1261] T e582869bb3104cb0943c01a4117cf2ea P 161d3b6a40d44954a10ffed41a6f0a16 [term 1 FOLLOWER]: Refusing update from remote peer 347c0427a88e45c5b1fad396bbe0f26a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:31.414193 3728 raft_consensus.cc:1261] T e582869bb3104cb0943c01a4117cf2ea P 60ef4cd12ace492e9b4fc35fed37d033 [term 1 FOLLOWER]: Refusing update from remote peer 347c0427a88e45c5b1fad396bbe0f26a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:31.414427 3817 consensus_queue.cc:1035] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a [LEADER]: Connected to new peer: Peer: permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:31.414538 3803 consensus_queue.cc:1035] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:31.415083 3805 tablet_replica.cc:439] T e582869bb3104cb0943c01a4117cf2ea P 161d3b6a40d44954a10ffed41a6f0a16: TxnStatusTablet state changed. Reason: New leader 347c0427a88e45c5b1fad396bbe0f26a. Latest consensus state: current_term: 1 leader_uuid: "347c0427a88e45c5b1fad396bbe0f26a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } } }
I20260621 05:20:31.415196 3805 tablet_replica.cc:442] T e582869bb3104cb0943c01a4117cf2ea P 161d3b6a40d44954a10ffed41a6f0a16: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:20:31.415405 3804 tablet_replica.cc:439] T e582869bb3104cb0943c01a4117cf2ea P 60ef4cd12ace492e9b4fc35fed37d033: TxnStatusTablet state changed. Reason: New leader 347c0427a88e45c5b1fad396bbe0f26a. Latest consensus state: current_term: 1 leader_uuid: "347c0427a88e45c5b1fad396bbe0f26a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } } }
I20260621 05:20:31.415501 3804 tablet_replica.cc:442] T e582869bb3104cb0943c01a4117cf2ea P 60ef4cd12ace492e9b4fc35fed37d033: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:20:31.415709 3807 tablet_replica.cc:439] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "347c0427a88e45c5b1fad396bbe0f26a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } } }
I20260621 05:20:31.415789 3807 tablet_replica.cc:442] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:20:31.416163 3805 tablet_replica.cc:439] T e582869bb3104cb0943c01a4117cf2ea P 161d3b6a40d44954a10ffed41a6f0a16: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "347c0427a88e45c5b1fad396bbe0f26a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } } }
I20260621 05:20:31.416245 3805 tablet_replica.cc:442] T e582869bb3104cb0943c01a4117cf2ea P 161d3b6a40d44954a10ffed41a6f0a16: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:20:31.417321 3803 tablet_replica.cc:439] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "347c0427a88e45c5b1fad396bbe0f26a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } } }
I20260621 05:20:31.417395 3803 tablet_replica.cc:442] T e582869bb3104cb0943c01a4117cf2ea P 347c0427a88e45c5b1fad396bbe0f26a: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:20:31.417798 3812 mvcc.cc:205] Tried to move back new op lower bound from 7299150771870785536 to 7299150771823542272. Current Snapshot: MvccSnapshot[applied={T|T < 7299150771870785536}]
I20260621 05:20:31.417905 3804 tablet_replica.cc:439] T e582869bb3104cb0943c01a4117cf2ea P 60ef4cd12ace492e9b4fc35fed37d033: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "347c0427a88e45c5b1fad396bbe0f26a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 } } peers { permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 } } peers { permanent_uuid: "60ef4cd12ace492e9b4fc35fed37d033" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45657 } } }
I20260621 05:20:31.417968 3804 tablet_replica.cc:442] T e582869bb3104cb0943c01a4117cf2ea P 60ef4cd12ace492e9b4fc35fed37d033: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:20:31.420521 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.417268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35128 (local address 127.2.32.3:45657)
0621 05:20:31.417310 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:31.417312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:31.417541 (+ 229us) server_negotiation.cc:407] Connection header received
0621 05:20:31.417609 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:31.417610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:31.417634 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:31.417720 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:31.418075 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.418604 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.419671 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.419862 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.419920 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.420074 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:31.420174 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:31.420230 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:31.420297 (+ 67us) server_negotiation.cc:299] Negotiation successful
0621 05:20:31.420315 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:20:31.425807 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.423496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:40465 (local address 127.2.32.2:41747)
0621 05:20:31.423554 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:31.423556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:31.423567 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:20:31.423589 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:31.423590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:31.423615 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:31.423689 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:31.423858 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.424361 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.425142 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.425325 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.425393 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.425572 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:31.425604 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:31.425662 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:31.425724 (+ 62us) server_negotiation.cc:299] Negotiation successful
0621 05:20:31.425743 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:20:31.425817 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.423395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:41747 (local address 127.2.32.3:40465)
0621 05:20:31.423485 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:31.423496 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:31.423544 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:31.423692 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:31.423692 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:31.423693 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:20:31.423770 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.423771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.424463 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.424464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.425031 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.425033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.425404 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.425405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.425466 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.425483 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:20:31.425664 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:20:31.425666 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:20:31.425721 (+ 55us) client_negotiation.cc:240] Negotiation successful
0621 05:20:31.425754 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 05:20:31.429062 3588 raft_consensus.cc:1261] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a [term 1 FOLLOWER]: Refusing update from remote peer 60ef4cd12ace492e9b4fc35fed37d033: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:31.429468 3448 raft_consensus.cc:1261] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16 [term 1 FOLLOWER]: Refusing update from remote peer 60ef4cd12ace492e9b4fc35fed37d033: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:31.429425 3804 consensus_queue.cc:1035] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "347c0427a88e45c5b1fad396bbe0f26a" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 41747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:31.429788 3804 consensus_queue.cc:1035] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "161d3b6a40d44954a10ffed41a6f0a16" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:31.438966 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.436067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:58185 (local address 127.2.32.3:45657)
0621 05:20:31.436105 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:31.436107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:31.436279 (+ 172us) server_negotiation.cc:407] Connection header received
0621 05:20:31.436311 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:31.436312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:31.436328 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:31.436400 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:31.436549 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.437077 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.438287 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.438488 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.438550 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.438743 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:31.438773 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:31.438829 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:31.438893 (+ 64us) server_negotiation.cc:299] Negotiation successful
0621 05:20:31.438914 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:20:31.439026 3851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.436012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:45657 (local address 127.2.32.2:58185)
0621 05:20:31.436151 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:31.436162 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:31.436222 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:31.436424 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:31.436425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:31.436426 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:20:31.436472 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.436473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.437162 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.437163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.438167 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:31.438170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.438568 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.438569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:31.438635 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.438649 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:20:31.438844 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:20:31.438846 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:20:31.438892 (+ 46us) client_negotiation.cc:240] Negotiation successful
0621 05:20:31.438939 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260621 05:20:31.449501 3290 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:20:31.449682 3290 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:20:31.450573 3290 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9ae6a6ba06042b6be64f749919c43bf: Sending DeleteTablet for 3 replicas of tablet 57b8dd93c6344dfe8d40fb3d908d74ad
I20260621 05:20:31.450886 3568 tablet_service.cc:1514] Processing DeleteTablet for tablet 57b8dd93c6344dfe8d40fb3d908d74ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:31 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:51446
I20260621 05:20:31.450994 3822 tablet_replica.cc:330] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a: stopping tablet replica
I20260621 05:20:31.450968 3708 tablet_service.cc:1514] Processing DeleteTablet for tablet 57b8dd93c6344dfe8d40fb3d908d74ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:31 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:35118
I20260621 05:20:31.450970 3428 tablet_service.cc:1514] Processing DeleteTablet for tablet 57b8dd93c6344dfe8d40fb3d908d74ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:31 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:37900
I20260621 05:20:31.451035 3822 raft_consensus.cc:2229] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:20:31.451071 3822 raft_consensus.cc:2258] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:31.451098 3821 tablet_replica.cc:330] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16: stopping tablet replica
I20260621 05:20:31.451184 3821 raft_consensus.cc:2229] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:20:31.451226 3821 raft_consensus.cc:2258] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:31.451311 3820 tablet_replica.cc:330] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033: stopping tablet replica
I20260621 05:20:31.451368 3820 raft_consensus.cc:2229] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:20:31.451432 3822 ts_tablet_manager.cc:1905] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:31.451469 3820 raft_consensus.cc:2258] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:31.451757 3821 ts_tablet_manager.cc:1905] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:31.453251 3822 ts_tablet_manager.cc:1918] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:20:31.453341 3822 log.cc:1196] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/57b8dd93c6344dfe8d40fb3d908d74ad
I20260621 05:20:31.453351 3821 ts_tablet_manager.cc:1918] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:20:31.453410 3821 log.cc:1196] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-0/wal/wals/57b8dd93c6344dfe8d40fb3d908d74ad
I20260621 05:20:31.453675 3821 ts_tablet_manager.cc:1939] T 57b8dd93c6344dfe8d40fb3d908d74ad P 161d3b6a40d44954a10ffed41a6f0a16: Deleting consensus metadata
I20260621 05:20:31.453753 3822 ts_tablet_manager.cc:1939] T 57b8dd93c6344dfe8d40fb3d908d74ad P 347c0427a88e45c5b1fad396bbe0f26a: Deleting consensus metadata
I20260621 05:20:31.454018 3278 catalog_manager.cc:4841] TS 161d3b6a40d44954a10ffed41a6f0a16 (127.2.32.1:37679): tablet 57b8dd93c6344dfe8d40fb3d908d74ad (table test-table [id=efdd3be218e64e7296bd0e2b0f5bb105]) successfully deleted
I20260621 05:20:31.454069 3277 catalog_manager.cc:4841] TS 347c0427a88e45c5b1fad396bbe0f26a (127.2.32.2:41747): tablet 57b8dd93c6344dfe8d40fb3d908d74ad (table test-table [id=efdd3be218e64e7296bd0e2b0f5bb105]) successfully deleted
I20260621 05:20:31.454332 3820 ts_tablet_manager.cc:1905] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:31.455979 3820 ts_tablet_manager.cc:1918] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:20:31.456036 3820 log.cc:1196] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/57b8dd93c6344dfe8d40fb3d908d74ad
I20260621 05:20:31.456251 3820 ts_tablet_manager.cc:1939] T 57b8dd93c6344dfe8d40fb3d908d74ad P 60ef4cd12ace492e9b4fc35fed37d033: Deleting consensus metadata
I20260621 05:20:31.456573 3275 catalog_manager.cc:4841] TS 60ef4cd12ace492e9b4fc35fed37d033 (127.2.32.3:45657): tablet 57b8dd93c6344dfe8d40fb3d908d74ad (table test-table [id=efdd3be218e64e7296bd0e2b0f5bb105]) successfully deleted
Jun 21 05:20:31 dist-test-slave-b357 krb5kdc[3246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019231, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.32.1@KRBTEST.COM
I20260621 05:20:31.464114 3795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.454260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37916 (local address 127.2.32.1:37679)
0621 05:20:31.454304 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:31.454306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:31.454441 (+ 135us) server_negotiation.cc:407] Connection header received
0621 05:20:31.454544 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:31.454545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:31.454555 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:31.454584 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:31.455340 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.456036 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.457115 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.457387 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.457437 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.460808 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:31.460816 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:31.460816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:31.460824 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:31.462752 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:31.463155 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:31.463158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:31.463159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:31.463234 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:31.463483 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:31.463484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:31.463484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:31.463632 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:31.463740 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:31.463943 (+ 203us) server_negotiation.cc:299] Negotiation successful
0621 05:20:31.464019 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260621 05:20:31.464633 3488 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37916
I20260621 05:20:31.475931 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:31.465522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59946 (local address 127.2.32.62:42743)
0621 05:20:31.465576 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:31.465578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:31.467283 (+ 1705us) server_negotiation.cc:407] Connection header received
0621 05:20:31.467385 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:31.467386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:31.467399 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:31.467466 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:31.468078 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.468850 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.470353 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:31.470662 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:31.470746 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:31.472787 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:31.472789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:31.472789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:31.472796 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:31.474536 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:31.474999 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:31.475000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:31.475001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:31.475060 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:31.475294 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:31.475295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:31.475295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:31.475429 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:31.475531 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:31.475719 (+ 188us) server_negotiation.cc:299] Negotiation successful
0621 05:20:31.475833 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260621 05:20:31.476114 3290 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59946
I20260621 05:20:31.476835 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 3363
I20260621 05:20:31.482877 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 3503
I20260621 05:20:31.488405 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 3643
I20260621 05:20:31.494099 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 3259
2026-06-21T05:20:31Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2324 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:31 dist-test-slave-b357 krb5kdc[3860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:31 dist-test-slave-b357 krb5kdc[3860](info): set up 2 sockets
Jun 21 05:20:31 dist-test-slave-b357 krb5kdc[3860](info): commencing operation
krb5kdc: starting...
W20260621 05:20:32.595554 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:20:32 dist-test-slave-b357 krb5kdc[3860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:20:32.751595 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:45741
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:32937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:45741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:32.843871 3873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:32.844498 3873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:32.844869 3873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:32.851282 3873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:32.851425 3873 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 05:20:32.851451 3873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:32.851475 3873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:32.851640 3873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:32.852048 3873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:32.854445 3873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:32937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:45741
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:45741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:32.854741 3873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:32.854998 3873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:32.860842 3879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:32.860700 3881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:32.861023 3873 server_base.cc:1034] running on GCE node
W20260621 05:20:32.860702 3878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:32.861390 3873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:32.862164 3873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:32.863444 3873 hybrid_clock.cc:648] HybridClock initialized: now 1782019232863399 us; error 81 us; skew 500 ppm
Jun 21 05:20:32 dist-test-slave-b357 krb5kdc[3860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019232, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:32.872663 3873 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:20:32.873183 3873 webserver.cc:460] Webserver started at http://127.2.32.62:34083/ using document root <none> and password file <none>
I20260621 05:20:32.873472 3873 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:32.873546 3873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:32.873703 3873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:32.874817 3873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "3c968bb12bff45e390b40346d62cf71f"
format_stamp: "Formatted at 2026-06-21 05:20:32 on dist-test-slave-b357"
server_key: "29a38331d0a8aa809f17ea62591c29e8"
server_key_iv: "b686dc34e1d3b0a27ae994d4c9e7938e"
server_key_version: "encryptionkey@0"
I20260621 05:20:32.875268 3873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "3c968bb12bff45e390b40346d62cf71f"
format_stamp: "Formatted at 2026-06-21 05:20:32 on dist-test-slave-b357"
server_key: "29a38331d0a8aa809f17ea62591c29e8"
server_key_iv: "b686dc34e1d3b0a27ae994d4c9e7938e"
server_key_version: "encryptionkey@0"
I20260621 05:20:32.876748 3873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:32.877652 3888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:32.877990 3873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:32.878090 3873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "3c968bb12bff45e390b40346d62cf71f"
format_stamp: "Formatted at 2026-06-21 05:20:32 on dist-test-slave-b357"
server_key: "29a38331d0a8aa809f17ea62591c29e8"
server_key_iv: "b686dc34e1d3b0a27ae994d4c9e7938e"
server_key_version: "encryptionkey@0"
I20260621 05:20:32.878175 3873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:32.915509 3873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:32.917970 3873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:32.918087 3873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:32.924292 3873 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:45741
I20260621 05:20:32.924515 3950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:45741 every 8 connection(s)
I20260621 05:20:32.924690 3873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:32.925958 3951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:32.928341 3951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f: Bootstrap starting.
I20260621 05:20:32.928830 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 3873
I20260621 05:20:32.928930 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:20:32.929159 3951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:32.929221 2176 external_mini_cluster.cc:1428] Setting key 0389a91bfa8280aab53dc048733603c2
I20260621 05:20:32.929510 3951 log.cc:824] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:32.930657 3951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f: No bootstrap required, opened a new log
I20260621 05:20:32.933360 3951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c968bb12bff45e390b40346d62cf71f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45741 } }
I20260621 05:20:32.933537 3951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:32.933569 3951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c968bb12bff45e390b40346d62cf71f, State: Initialized, Role: FOLLOWER
I20260621 05:20:32.933691 3951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c968bb12bff45e390b40346d62cf71f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45741 } }
I20260621 05:20:32.933791 3951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:32.933840 3951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:32.933892 3951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:32.934451 3951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c968bb12bff45e390b40346d62cf71f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45741 } }
I20260621 05:20:32.934587 3951 leader_election.cc:302] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c968bb12bff45e390b40346d62cf71f; no voters:
I20260621 05:20:32.934733 3951 leader_election.cc:288] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:32.934808 3956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:32.934984 3956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f [term 1 LEADER]: Becoming Leader. State: Replica: 3c968bb12bff45e390b40346d62cf71f, State: Running, Role: LEADER
I20260621 05:20:32.935029 3951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:32.935070 3956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c968bb12bff45e390b40346d62cf71f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45741 } }
I20260621 05:20:32.935964 3956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c968bb12bff45e390b40346d62cf71f. Latest consensus state: current_term: 1 leader_uuid: "3c968bb12bff45e390b40346d62cf71f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c968bb12bff45e390b40346d62cf71f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45741 } } }
I20260621 05:20:32.936048 3956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:32.936231 3957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c968bb12bff45e390b40346d62cf71f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c968bb12bff45e390b40346d62cf71f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45741 } } }
I20260621 05:20:32.936264 3966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:32.936290 3957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:32.936846 3966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 21 05:20:32 dist-test-slave-b357 krb5kdc[3860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:32.938859 3966 catalog_manager.cc:1347] Generated new cluster ID: d7307bf53bbf4deeba5c90ee32d92deb
I20260621 05:20:32.938910 3966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:32.945137 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:32.930125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52732 (local address 127.2.32.62:45741)
0621 05:20:32.930401 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:32.930411 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:32.930453 (+ 42us) server_negotiation.cc:407] Connection header received
0621 05:20:32.931081 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:32.931089 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:32.931182 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:32.931436 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:32.932071 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:32.933618 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:32.934525 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:32.935179 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:32.935637 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:32.938690 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:32.938697 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:32.938697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:32.938876 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:32.942510 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:32.943009 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:32.943012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:32.943013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:32.943116 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:32.943392 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:32.943393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:32.943394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:32.943775 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:32.944315 (+ 540us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:32.944389 (+ 74us) server_negotiation.cc:299] Negotiation successful
0621 05:20:32.944529 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":138,"threads_started":1}
I20260621 05:20:32.963441 3966 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:32.965368 3966 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:32.971300 3966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f: Generated new TSK 0
I20260621 05:20:32.971660 3966 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
I20260621 05:20:33.007781 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:45741
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:32937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:33.098695 3977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:33.098910 3977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:33.098937 3977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:33.100211 3977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:33.100262 3977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:33.100283 3977 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 05:20:33.100296 3977 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:20:33.100313 3977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:20:33.101794 3977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:32937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.32.62:45741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:33.102043 3977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:33.102298 3977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:33.108337 3985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:33.108337 3982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:33.108381 3983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:33.108732 3977 server_base.cc:1034] running on GCE node
I20260621 05:20:33.108980 3977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:33.109277 3977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:33.110487 3977 hybrid_clock.cc:648] HybridClock initialized: now 1782019233110459 us; error 52 us; skew 500 ppm
Jun 21 05:20:33 dist-test-slave-b357 krb5kdc[3860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019233, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:33.113210 3977 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:20:33.113682 3977 webserver.cc:460] Webserver started at http://127.2.32.1:40395/ using document root <none> and password file <none>
I20260621 05:20:33.113926 3977 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:33.114005 3977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:33.114157 3977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:33.115279 3977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "be1ed1d9dd334995b627a6cddfcf9c30"
format_stamp: "Formatted at 2026-06-21 05:20:33 on dist-test-slave-b357"
server_key: "28355f4c098e8cdbb73a6a155ea877b8"
server_key_iv: "87ba2ae03384de0e38937ee75f608bf9"
server_key_version: "encryptionkey@0"
I20260621 05:20:33.115638 3977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "be1ed1d9dd334995b627a6cddfcf9c30"
format_stamp: "Formatted at 2026-06-21 05:20:33 on dist-test-slave-b357"
server_key: "28355f4c098e8cdbb73a6a155ea877b8"
server_key_iv: "87ba2ae03384de0e38937ee75f608bf9"
server_key_version: "encryptionkey@0"
I20260621 05:20:33.117247 3977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:20:33.118371 3992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:33.118765 3977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:33.118976 3977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "be1ed1d9dd334995b627a6cddfcf9c30"
format_stamp: "Formatted at 2026-06-21 05:20:33 on dist-test-slave-b357"
server_key: "28355f4c098e8cdbb73a6a155ea877b8"
server_key_iv: "87ba2ae03384de0e38937ee75f608bf9"
server_key_version: "encryptionkey@0"
I20260621 05:20:33.119063 3977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:33.147308 3977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:33.150127 3977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:33.150362 3977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:33.151218 3977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:33.151293 3977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:33.151364 3977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:33.151400 3977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 05:20:33 dist-test-slave-b357 krb5kdc[3860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019233, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:33.159333 3977 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:45881
I20260621 05:20:33.159431 4107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:45881 every 8 connection(s)
I20260621 05:20:33.159780 3977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:20:33.162132 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.152080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:59523 (local address 127.2.32.62:45741)
0621 05:20:33.152173 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.152176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.152424 (+ 248us) server_negotiation.cc:407] Connection header received
0621 05:20:33.153253 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.153255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.153270 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.153358 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:33.154205 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.154729 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.155895 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.156115 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.156222 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.158936 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:33.158937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:33.158938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:33.158945 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:33.160638 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:33.161088 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:33.161089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:33.161089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:33.161150 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:33.161459 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:33.161460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:33.161460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:33.161589 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:33.161760 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.161985 (+ 225us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.162067 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 05:20:33.162220 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.152031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45741 (local address 127.2.32.1:59523)
0621 05:20:33.152272 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:33.152316 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:33.153075 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:33.153551 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:33.153556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:33.153899 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:33.154099 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.154104 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.154822 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.154823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.155792 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.155794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.156231 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.156232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.156401 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.157021 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:33.157034 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:33.158833 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:33.160725 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:33.160731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:33.160733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:33.160967 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:33.161249 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:33.161250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:33.161250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:33.161354 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:33.161683 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:33.161689 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:33.161806 (+ 117us) client_negotiation.cc:769] Sending connection context
0621 05:20:33.161926 (+ 120us) client_negotiation.cc:240] Negotiation successful
0621 05:20:33.162048 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":153,"threads_started":1}
I20260621 05:20:33.162730 4108 heartbeater.cc:347] Connected to a master server at 127.2.32.62:45741
I20260621 05:20:33.162811 4108 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:33.162984 4108 heartbeater.cc:511] Master 127.2.32.62:45741 requested a full tablet report, sending...
I20260621 05:20:33.163513 3905 ts_manager.cc:194] Registered new tserver with Master: be1ed1d9dd334995b627a6cddfcf9c30 (127.2.32.1:45881)
I20260621 05:20:33.163921 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 3977
I20260621 05:20:33.164037 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:20:33.164227 2176 external_mini_cluster.cc:1428] Setting key 021f756623a4a6f19d10403f74825d92
I20260621 05:20:33.164404 3905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:59523
I20260621 05:20:33.173820 4114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.163789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45741 (local address 127.2.32.1:60525)
0621 05:20:33.163933 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:33.163944 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:33.163998 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:33.164659 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:33.164660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:33.164851 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:33.164939 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.164940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.166715 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.166717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.167521 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.167524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.168137 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.168139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.168248 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.169134 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:33.169139 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:33.169533 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:33.172462 (+ 2929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:33.172464 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:33.172464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:33.172663 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:33.172947 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:33.172948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:33.172948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:33.173016 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:33.173377 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:33.173379 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:33.173443 (+ 64us) client_negotiation.cc:769] Sending connection context
0621 05:20:33.173529 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 05:20:33.173719 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":9,"thread_start_us":75,"threads_started":1}
I20260621 05:20:33.173836 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.164480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:60525 (local address 127.2.32.62:45741)
0621 05:20:33.164520 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.164522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.164531 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:20:33.164550 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.164550 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.164559 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.164615 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:33.165002 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.166396 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.167658 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.168019 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.168102 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.169730 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:33.169732 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:33.169733 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:33.169750 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:33.172307 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:33.172784 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:33.172785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:33.172786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:33.172851 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:33.173096 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:33.173097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:33.173098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:33.173254 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:33.173334 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.173599 (+ 265us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.173700 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for kudu/127.2.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
I20260621 05:20:33.198292 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:45741
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:32937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:33.293416 4117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:33.293624 4117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:33.293653 4117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:33.294739 4117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:33.294781 4117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:33.294804 4117 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 05:20:33.294818 4117 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:20:33.294836 4117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:20:33.296630 4117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:32937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.32.62:45741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:33.296932 4117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:33.297170 4117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:33.302839 4125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:33.302830 4122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:33.303166 4117 server_base.cc:1034] running on GCE node
W20260621 05:20:33.302852 4123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:33.303548 4117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:33.303961 4117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:33.305166 4117 hybrid_clock.cc:648] HybridClock initialized: now 1782019233305112 us; error 75 us; skew 500 ppm
Jun 21 05:20:33 dist-test-slave-b357 krb5kdc[3860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019233, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:33.308318 4117 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:20:33.309037 4117 webserver.cc:460] Webserver started at http://127.2.32.2:36875/ using document root <none> and password file <none>
I20260621 05:20:33.309357 4117 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:33.309414 4117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:33.309571 4117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:33.310832 4117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "35c712dea5ca49b4a3da080f62df9adb"
format_stamp: "Formatted at 2026-06-21 05:20:33 on dist-test-slave-b357"
server_key: "f8fc70ef03181431664b70e063998680"
server_key_iv: "c3c3e19697decb0b6f9daa4764b05edd"
server_key_version: "encryptionkey@0"
I20260621 05:20:33.311286 4117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "35c712dea5ca49b4a3da080f62df9adb"
format_stamp: "Formatted at 2026-06-21 05:20:33 on dist-test-slave-b357"
server_key: "f8fc70ef03181431664b70e063998680"
server_key_iv: "c3c3e19697decb0b6f9daa4764b05edd"
server_key_version: "encryptionkey@0"
I20260621 05:20:33.313045 4117 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 05:20:33.313907 4132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:33.314162 4117 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:33.314266 4117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "35c712dea5ca49b4a3da080f62df9adb"
format_stamp: "Formatted at 2026-06-21 05:20:33 on dist-test-slave-b357"
server_key: "f8fc70ef03181431664b70e063998680"
server_key_iv: "c3c3e19697decb0b6f9daa4764b05edd"
server_key_version: "encryptionkey@0"
I20260621 05:20:33.314356 4117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:33.332568 4117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:33.335431 4117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:33.335639 4117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:33.336290 4117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:33.336354 4117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:33.336391 4117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:33.336428 4117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:33.344373 4117 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:37907
I20260621 05:20:33.344457 4247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:37907 every 8 connection(s)
I20260621 05:20:33.344810 4117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:20:33 dist-test-slave-b357 krb5kdc[3860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019233, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:33.350570 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.337283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:48261 (local address 127.2.32.62:45741)
0621 05:20:33.337353 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.337356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.337575 (+ 219us) server_negotiation.cc:407] Connection header received
0621 05:20:33.338823 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.338824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.338839 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.338904 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:33.340337 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.341206 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.342776 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.343081 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.343178 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.346184 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:33.346186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:33.346186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:33.346192 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:33.348073 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:33.349034 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:33.349036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:33.349036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:33.349156 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:33.349649 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:33.349651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:33.349651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:33.349812 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:33.349893 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.350319 (+ 426us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.350491 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 05:20:33.350620 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.337234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45741 (local address 127.2.32.2:48261)
0621 05:20:33.337432 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:33.337468 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:33.338647 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:33.339017 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:33.339024 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:33.339558 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:33.340037 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.340099 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.341314 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.341315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.342670 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.342673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.343214 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.343216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.343396 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.344008 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:33.344019 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:33.346078 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:33.348418 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:33.348429 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:33.348432 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:33.348777 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:33.349281 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:33.349283 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:33.349283 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:33.349422 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:33.349897 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:33.349903 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:33.350056 (+ 153us) client_negotiation.cc:769] Sending connection context
0621 05:20:33.350154 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 05:20:33.350389 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":107,"threads_started":1}
I20260621 05:20:33.351416 4248 heartbeater.cc:347] Connected to a master server at 127.2.32.62:45741
I20260621 05:20:33.351509 4248 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:33.351696 4248 heartbeater.cc:511] Master 127.2.32.62:45741 requested a full tablet report, sending...
I20260621 05:20:33.352277 3905 ts_manager.cc:194] Registered new tserver with Master: 35c712dea5ca49b4a3da080f62df9adb (127.2.32.2:37907)
I20260621 05:20:33.352785 3905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:48261
I20260621 05:20:33.354079 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 4117
I20260621 05:20:33.354281 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:20:33.354470 2176 external_mini_cluster.cc:1428] Setting key d2d65ac529323e1b4c615aca49b3acaa
I20260621 05:20:33.360011 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.352620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:45463 (local address 127.2.32.62:45741)
0621 05:20:33.352686 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.352687 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.352803 (+ 116us) server_negotiation.cc:407] Connection header received
0621 05:20:33.352826 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.352827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.352836 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.352930 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:33.353547 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.354091 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.355180 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.355500 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.355656 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.356823 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:33.356825 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:33.356825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:33.356832 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:33.358595 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:33.358963 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:33.358965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:33.358965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:33.359022 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:33.359400 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:33.359401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:33.359402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:33.359562 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:33.359641 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.359827 (+ 186us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.359911 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 05:20:33.360020 4254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.352532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45741 (local address 127.2.32.2:45463)
0621 05:20:33.352686 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:33.352697 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:33.352758 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:33.352985 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:33.352986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:33.353191 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:33.353314 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.353319 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.354225 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.354226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.355061 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.355063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.355622 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.355623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.355718 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.356383 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:33.356388 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:33.356669 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:33.358698 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:33.358699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:33.358699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:33.358870 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:33.359087 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:33.359088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:33.359088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:33.359320 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:33.359646 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:33.359647 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:33.359691 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 05:20:33.359795 (+ 104us) client_negotiation.cc:240] Negotiation successful
0621 05:20:33.359909 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.2.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
I20260621 05:20:33.395843 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:45741
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:32937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:33.489550 4257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:33.489754 4257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:33.489778 4257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:33.490867 4257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:33.490904 4257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:33.490924 4257 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 05:20:33.490936 4257 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:20:33.490953 4257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:20:33.492396 4257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:32937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.32.62:45741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:33.492589 4257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:33.492758 4257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:33.497896 4265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:33.497941 4262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:33.498008 4263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:33.498656 4257 server_base.cc:1034] running on GCE node
I20260621 05:20:33.498878 4257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:33.499161 4257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:33.500334 4257 hybrid_clock.cc:648] HybridClock initialized: now 1782019233500320 us; error 30 us; skew 500 ppm
Jun 21 05:20:33 dist-test-slave-b357 krb5kdc[3860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019233, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:33.503079 4257 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:20:33.503587 4257 webserver.cc:460] Webserver started at http://127.2.32.3:45177/ using document root <none> and password file <none>
I20260621 05:20:33.503822 4257 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:33.503891 4257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:33.504042 4257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:33.505113 4257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "66e512a22ffa4800926686468233a6c3"
format_stamp: "Formatted at 2026-06-21 05:20:33 on dist-test-slave-b357"
server_key: "7c4ac10c539673e5136fc7397fb01297"
server_key_iv: "98b4f14f0e39747ebc08da7b5af45ab7"
server_key_version: "encryptionkey@0"
I20260621 05:20:33.505492 4257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "66e512a22ffa4800926686468233a6c3"
format_stamp: "Formatted at 2026-06-21 05:20:33 on dist-test-slave-b357"
server_key: "7c4ac10c539673e5136fc7397fb01297"
server_key_iv: "98b4f14f0e39747ebc08da7b5af45ab7"
server_key_version: "encryptionkey@0"
I20260621 05:20:33.506997 4257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260621 05:20:33.508266 4272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:33.508591 4257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:33.508730 4257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "66e512a22ffa4800926686468233a6c3"
format_stamp: "Formatted at 2026-06-21 05:20:33 on dist-test-slave-b357"
server_key: "7c4ac10c539673e5136fc7397fb01297"
server_key_iv: "98b4f14f0e39747ebc08da7b5af45ab7"
server_key_version: "encryptionkey@0"
I20260621 05:20:33.508837 4257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:33.531486 4257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:33.534236 4257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:33.534430 4257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:33.535153 4257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:33.535203 4257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:33.535239 4257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:33.535279 4257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 05:20:33 dist-test-slave-b357 krb5kdc[3860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019233, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:33.542584 4257 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:45001
I20260621 05:20:33.542691 4387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:45001 every 8 connection(s)
I20260621 05:20:33.542968 4257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:20:33.546056 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.535557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45741 (local address 127.2.32.3:43309)
0621 05:20:33.535727 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:33.535766 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:33.536431 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:33.536677 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:33.536682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:33.536939 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:33.537126 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.537131 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.537822 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.537823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.538654 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.538656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.539044 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.539044 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.539214 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.539775 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:33.539786 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:33.542429 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:33.544261 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:33.544272 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:33.544275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:33.544621 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:33.544927 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:33.544928 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:33.544928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:33.545108 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:33.545509 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:33.545515 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:33.545648 (+ 133us) client_negotiation.cc:769] Sending connection context
0621 05:20:33.545758 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 05:20:33.545899 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":102,"threads_started":1}
I20260621 05:20:33.546015 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.535513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:43309 (local address 127.2.32.62:45741)
0621 05:20:33.535578 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.535581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.535877 (+ 296us) server_negotiation.cc:407] Connection header received
0621 05:20:33.536541 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.536542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.536555 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.536595 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:33.537217 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.537733 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.538752 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.538952 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.539032 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.542579 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:33.542581 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:33.542581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:33.542588 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:33.544138 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:33.544757 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:33.544758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:33.544759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:33.544818 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:33.545234 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:33.545236 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:33.545236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:33.545407 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:33.545483 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.545834 (+ 351us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.545930 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:20:33.546666 4388 heartbeater.cc:347] Connected to a master server at 127.2.32.62:45741
I20260621 05:20:33.546792 4388 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:33.546991 4388 heartbeater.cc:511] Master 127.2.32.62:45741 requested a full tablet report, sending...
I20260621 05:20:33.547513 3904 ts_manager.cc:194] Registered new tserver with Master: 66e512a22ffa4800926686468233a6c3 (127.2.32.3:45001)
I20260621 05:20:33.547933 3904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:43309
I20260621 05:20:33.552317 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 4257
I20260621 05:20:33.552451 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:20:33.552712 2176 external_mini_cluster.cc:1428] Setting key 5660eb2679bc59cf3945ed13559a38bd
I20260621 05:20:33.554276 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:20:33.555349 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.548338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:38959 (local address 127.2.32.62:45741)
0621 05:20:33.548393 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.548395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.548568 (+ 173us) server_negotiation.cc:407] Connection header received
0621 05:20:33.548601 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.548601 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.548612 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.548675 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:33.549082 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.549771 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.550578 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.550811 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.550882 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.552124 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:33.552126 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:33.552127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:33.552134 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:33.554164 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:33.554556 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:33.554557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:33.554557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:33.554614 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:33.554814 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:33.554815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:33.554815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:33.554926 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:33.554979 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.555205 (+ 226us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.555284 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260621 05:20:33.555445 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.548235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45741 (local address 127.2.32.3:38959)
0621 05:20:33.548467 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:33.548480 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:33.548554 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:33.548698 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:33.548699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:33.548901 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:33.548959 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.548961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.549928 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.549929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.550474 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.550476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.550906 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.550907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.550961 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.551709 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:33.551713 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:33.551986 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:33.554268 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:33.554269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:33.554269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:33.554471 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:33.554682 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:33.554683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:33.554683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:33.554749 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:33.554993 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:33.554994 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:33.555041 (+ 47us) client_negotiation.cc:769] Sending connection context
0621 05:20:33.555168 (+ 127us) client_negotiation.cc:240] Negotiation successful
0621 05:20:33.555285 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":147,"threads_started":1}
Jun 21 05:20:33 dist-test-slave-b357 krb5kdc[3860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 05:20:33 dist-test-slave-b357 krb5kdc[3860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:33.579757 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.571675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.2.32.62:45741)
0621 05:20:33.571749 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.571750 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.571781 (+ 31us) server_negotiation.cc:407] Connection header received
0621 05:20:33.571803 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.571803 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.571811 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.571861 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:33.572403 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.573061 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.574123 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.574305 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.574369 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.577127 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:33.577129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:33.577129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:33.577135 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:33.578592 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:33.578945 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:33.578946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:33.578946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:33.579001 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:33.579250 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:33.579250 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:33.579251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:33.579374 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:33.579453 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.579631 (+ 178us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.579694 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 05:20:33.582036 3904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:20:33.584220 3904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:20:33.591310 4407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.587067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:37907 (local address 127.0.0.1:54786)
0621 05:20:33.587544 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:33.587555 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:33.587599 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:33.587971 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:33.587972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:33.587973 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:33.588018 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.588019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.588888 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.588889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.590188 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.590191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.590934 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.590935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.591013 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.591027 (+ 14us) client_negotiation.cc:769] Sending connection context
0621 05:20:33.591161 (+ 134us) client_negotiation.cc:240] Negotiation successful
0621 05:20:33.591217 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":24,"thread_start_us":136,"threads_started":1}
I20260621 05:20:33.591325 4410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.587369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54786 (local address 127.2.32.2:37907)
0621 05:20:33.587582 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.587586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.587597 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:20:33.587696 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.587699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.587714 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.587807 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:33.588092 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.588800 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.590290 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.590826 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.590931 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.590992 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.591174 (+ 182us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.591216 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":94,"threads_started":1}
I20260621 05:20:33.591679 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.587353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39226 (local address 127.2.32.3:45001)
0621 05:20:33.587526 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.587531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.587543 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:20:33.587587 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.587589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.587603 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.587795 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:33.587965 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.589073 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.590326 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.591064 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.591375 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.591434 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.591469 (+ 35us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.591502 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":107,"threads_started":1}
I20260621 05:20:33.591769 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.587067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:45001 (local address 127.0.0.1:39226)
0621 05:20:33.587336 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:33.587363 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:33.587436 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:33.587795 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:33.587798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:33.587800 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:33.587885 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.587887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.589163 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.589163 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.590234 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.590235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.591195 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.591196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.591253 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.591259 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:20:33.591289 (+ 30us) client_negotiation.cc:240] Negotiation successful
0621 05:20:33.591332 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":139,"threads_started":1}
I20260621 05:20:33.592151 4408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.587114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.2.32.1:45881)
0621 05:20:33.587673 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.587678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.587713 (+ 35us) server_negotiation.cc:407] Connection header received
0621 05:20:33.587936 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.587938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.587955 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.588042 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:33.588236 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.589258 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.590394 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.591079 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.591897 (+ 818us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.591957 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.591988 (+ 31us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.592039 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":152,"threads_started":1}
I20260621 05:20:33.592301 4406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.586986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:45881 (local address 127.0.0.1:47050)
0621 05:20:33.587276 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:33.587668 (+ 392us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:33.587702 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:33.588100 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:33.588101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:33.588102 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:33.588142 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.588144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.589344 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.589346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.590310 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.590312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.591182 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.591183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.591252 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.591256 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:20:33.591295 (+ 39us) client_negotiation.cc:240] Negotiation successful
0621 05:20:33.591332 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":59,"threads_started":1}
I20260621 05:20:33.592166 4182 tablet_service.cc:1467] Processing CreateTablet for tablet ac8d0a06d86a42be9a16872a7d599afb (DEFAULT_TABLE table=test-table [id=d49744b708a9462ba1cb9d80127b3c3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:33.592375 4322 tablet_service.cc:1467] Processing CreateTablet for tablet ac8d0a06d86a42be9a16872a7d599afb (DEFAULT_TABLE table=test-table [id=d49744b708a9462ba1cb9d80127b3c3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:33.592547 4182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac8d0a06d86a42be9a16872a7d599afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:33.592653 4322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac8d0a06d86a42be9a16872a7d599afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:33.592795 4042 tablet_service.cc:1467] Processing CreateTablet for tablet ac8d0a06d86a42be9a16872a7d599afb (DEFAULT_TABLE table=test-table [id=d49744b708a9462ba1cb9d80127b3c3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:33.593017 4042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac8d0a06d86a42be9a16872a7d599afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:33.595875 4411 tablet_bootstrap.cc:492] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30: Bootstrap starting.
I20260621 05:20:33.596338 4412 tablet_bootstrap.cc:492] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3: Bootstrap starting.
I20260621 05:20:33.596553 4411 tablet_bootstrap.cc:654] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:33.596868 4411 log.cc:824] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:33.597175 4412 tablet_bootstrap.cc:654] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:33.597498 4411 tablet_bootstrap.cc:492] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30: No bootstrap required, opened a new log
I20260621 05:20:33.597551 4411 ts_tablet_manager.cc:1397] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:20:33.597642 4412 log.cc:824] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:33.597983 4413 tablet_bootstrap.cc:492] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb: Bootstrap starting.
I20260621 05:20:33.598376 4412 tablet_bootstrap.cc:492] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3: No bootstrap required, opened a new log
I20260621 05:20:33.598443 4412 ts_tablet_manager.cc:1397] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:20:33.598613 4413 tablet_bootstrap.cc:654] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:33.598937 4413 log.cc:824] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:33.599589 4411 raft_consensus.cc:348] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.599697 4413 tablet_bootstrap.cc:492] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb: No bootstrap required, opened a new log
I20260621 05:20:33.599735 4411 raft_consensus.cc:374] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:33.599757 4413 ts_tablet_manager.cc:1397] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:20:33.599785 4411 raft_consensus.cc:729] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1ed1d9dd334995b627a6cddfcf9c30, State: Initialized, Role: FOLLOWER
I20260621 05:20:33.599891 4411 consensus_queue.cc:260] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.600167 4411 ts_tablet_manager.cc:1428] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:20:33.600176 4108 heartbeater.cc:503] Master 127.2.32.62:45741 was elected leader, sending a full tablet report...
I20260621 05:20:33.600519 4412 raft_consensus.cc:348] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.600670 4412 raft_consensus.cc:374] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:33.600714 4412 raft_consensus.cc:729] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e512a22ffa4800926686468233a6c3, State: Initialized, Role: FOLLOWER
I20260621 05:20:33.600822 4412 consensus_queue.cc:260] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.601070 4412 ts_tablet_manager.cc:1428] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:20:33.601099 4388 heartbeater.cc:503] Master 127.2.32.62:45741 was elected leader, sending a full tablet report...
I20260621 05:20:33.602373 4413 raft_consensus.cc:348] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.602839 4413 raft_consensus.cc:374] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:33.602923 4413 raft_consensus.cc:729] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35c712dea5ca49b4a3da080f62df9adb, State: Initialized, Role: FOLLOWER
I20260621 05:20:33.603048 4413 consensus_queue.cc:260] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.603385 4413 ts_tablet_manager.cc:1428] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260621 05:20:33.603406 4248 heartbeater.cc:503] Master 127.2.32.62:45741 was elected leader, sending a full tablet report...
I20260621 05:20:33.628717 4419 raft_consensus.cc:482] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:20:33.628837 4419 raft_consensus.cc:504] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.629281 4419 leader_election.cc:288] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be1ed1d9dd334995b627a6cddfcf9c30 (127.2.32.1:45881), 66e512a22ffa4800926686468233a6c3 (127.2.32.3:45001)
I20260621 05:20:33.632702 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.629414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:45881 (local address 127.2.32.2:60805)
0621 05:20:33.629497 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:33.629521 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:33.629580 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:33.629854 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:33.629856 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:33.629858 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:33.629961 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.629964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.630814 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.630815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.631783 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.631785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.632446 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.632447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.632527 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.632531 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:20:33.632593 (+ 62us) client_negotiation.cc:240] Negotiation successful
0621 05:20:33.632630 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260621 05:20:33.632855 4408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.629484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:60805 (local address 127.2.32.1:45881)
0621 05:20:33.629563 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.629567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.629578 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:20:33.629703 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.629705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.629722 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.629787 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:33.630133 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.630666 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.631865 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.632345 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.632400 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.632408 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.632683 (+ 275us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.632765 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 05:20:33.633287 4062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8d0a06d86a42be9a16872a7d599afb" candidate_uuid: "35c712dea5ca49b4a3da080f62df9adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" is_pre_election: true
I20260621 05:20:33.633364 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.629668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:46315 (local address 127.2.32.3:45001)
0621 05:20:33.629810 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.629812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.629826 (+ 14us) server_negotiation.cc:407] Connection header received
0621 05:20:33.629860 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.629861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.629876 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.629962 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:33.630243 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.631005 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.632191 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.632830 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.632896 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.632904 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.633240 (+ 336us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.633310 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260621 05:20:33.633463 4062 raft_consensus.cc:2454] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35c712dea5ca49b4a3da080f62df9adb in term 0.
I20260621 05:20:33.633636 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.629481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:45001 (local address 127.2.32.2:46315)
0621 05:20:33.629700 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:33.629712 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:33.629802 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:33.629997 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:33.629998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:33.629999 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:33.630125 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.630127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.631110 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.631111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.632086 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.632089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.632955 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.632957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.633083 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.633129 (+ 46us) client_negotiation.cc:769] Sending connection context
0621 05:20:33.633473 (+ 344us) client_negotiation.cc:240] Negotiation successful
0621 05:20:33.633536 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":112,"threads_started":1}
I20260621 05:20:33.633720 4136 leader_election.cc:302] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35c712dea5ca49b4a3da080f62df9adb, be1ed1d9dd334995b627a6cddfcf9c30; no voters:
I20260621 05:20:33.633895 4419 raft_consensus.cc:2781] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:20:33.633970 4419 raft_consensus.cc:482] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:20:33.633999 4419 raft_consensus.cc:3037] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:33.634060 4342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8d0a06d86a42be9a16872a7d599afb" candidate_uuid: "35c712dea5ca49b4a3da080f62df9adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e512a22ffa4800926686468233a6c3" is_pre_election: true
I20260621 05:20:33.634259 4342 raft_consensus.cc:2454] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35c712dea5ca49b4a3da080f62df9adb in term 0.
I20260621 05:20:33.634876 4419 raft_consensus.cc:504] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.635156 4419 leader_election.cc:288] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [CANDIDATE]: Term 1 election: Requested vote from peers be1ed1d9dd334995b627a6cddfcf9c30 (127.2.32.1:45881), 66e512a22ffa4800926686468233a6c3 (127.2.32.3:45001)
I20260621 05:20:33.635380 4062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8d0a06d86a42be9a16872a7d599afb" candidate_uuid: "35c712dea5ca49b4a3da080f62df9adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1ed1d9dd334995b627a6cddfcf9c30"
I20260621 05:20:33.635334 4342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8d0a06d86a42be9a16872a7d599afb" candidate_uuid: "35c712dea5ca49b4a3da080f62df9adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e512a22ffa4800926686468233a6c3"
I20260621 05:20:33.635465 4062 raft_consensus.cc:3037] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:33.635460 4342 raft_consensus.cc:3037] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:33.636314 4062 raft_consensus.cc:2454] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35c712dea5ca49b4a3da080f62df9adb in term 1.
I20260621 05:20:33.636314 4342 raft_consensus.cc:2454] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35c712dea5ca49b4a3da080f62df9adb in term 1.
I20260621 05:20:33.636807 4135 leader_election.cc:302] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35c712dea5ca49b4a3da080f62df9adb, 66e512a22ffa4800926686468233a6c3; no voters:
I20260621 05:20:33.636961 4419 raft_consensus.cc:2781] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:33.637240 4419 raft_consensus.cc:686] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [term 1 LEADER]: Becoming Leader. State: Replica: 35c712dea5ca49b4a3da080f62df9adb, State: Running, Role: LEADER
I20260621 05:20:33.637465 4419 consensus_queue.cc:237] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.638634 3904 catalog_manager.cc:5495] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb reported cstate change: term changed from 0 to 1, leader changed from <none> to 35c712dea5ca49b4a3da080f62df9adb (127.2.32.2). New cstate: current_term: 1 leader_uuid: "35c712dea5ca49b4a3da080f62df9adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } health_report { overall_health: UNKNOWN } } }
W20260621 05:20:33.660698 4109 tablet.cc:2369] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 21 05:20:33 dist-test-slave-b357 krb5kdc[3860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019232, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:33.679764 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.672051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52752 (local address 127.2.32.62:45741)
0621 05:20:33.672172 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.672174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.672285 (+ 111us) server_negotiation.cc:407] Connection header received
0621 05:20:33.672386 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.672387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.672396 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.672467 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:33.672901 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.673383 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.674094 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.674276 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.674349 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.676856 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:33.676857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:33.676858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:33.676865 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:33.678538 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:33.678939 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:33.678939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:33.678940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:33.678997 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:33.679223 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:33.679223 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:33.679224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:33.679328 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:33.679403 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.679554 (+ 151us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.679652 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260621 05:20:33.679832 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.671985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45741 (local address 127.0.0.1:52752)
0621 05:20:33.672197 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:33.672216 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:33.672304 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:33.672470 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:33.672471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:33.672771 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:33.672817 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.672819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.673461 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.673462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.674014 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.674015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.674356 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.674357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.674413 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.675054 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:33.675059 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:33.676746 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:33.678667 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:33.678674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:33.678676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:33.678860 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:33.679063 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:33.679064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:33.679065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:33.679165 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:33.679394 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:33.679401 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:33.679445 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 05:20:33.679530 (+ 85us) client_negotiation.cc:240] Negotiation successful
0621 05:20:33.679710 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":81,"thread_start_us":99,"threads_started":1}
I20260621 05:20:33.682627 3904 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:52752:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 05:20:33.682941 3904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:20:33.685153 4322 tablet_service.cc:1467] Processing CreateTablet for tablet 33955ad97df447d387fb4cbb55698c23 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1b02916cd93f482ab3c139ecfdea14ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:20:33.685125 4042 tablet_service.cc:1467] Processing CreateTablet for tablet 33955ad97df447d387fb4cbb55698c23 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1b02916cd93f482ab3c139ecfdea14ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:20:33.685343 4042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33955ad97df447d387fb4cbb55698c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:33.685343 4322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33955ad97df447d387fb4cbb55698c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:33.686976 4412 tablet_bootstrap.cc:492] T 33955ad97df447d387fb4cbb55698c23 P 66e512a22ffa4800926686468233a6c3: Bootstrap starting.
I20260621 05:20:33.686976 4411 tablet_bootstrap.cc:492] T 33955ad97df447d387fb4cbb55698c23 P be1ed1d9dd334995b627a6cddfcf9c30: Bootstrap starting.
I20260621 05:20:33.687129 4182 tablet_service.cc:1467] Processing CreateTablet for tablet 33955ad97df447d387fb4cbb55698c23 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1b02916cd93f482ab3c139ecfdea14ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:20:33.687323 4182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33955ad97df447d387fb4cbb55698c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:33.687678 4411 tablet_bootstrap.cc:654] T 33955ad97df447d387fb4cbb55698c23 P be1ed1d9dd334995b627a6cddfcf9c30: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:33.687868 4412 tablet_bootstrap.cc:654] T 33955ad97df447d387fb4cbb55698c23 P 66e512a22ffa4800926686468233a6c3: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:33.688532 4411 tablet_bootstrap.cc:492] T 33955ad97df447d387fb4cbb55698c23 P be1ed1d9dd334995b627a6cddfcf9c30: No bootstrap required, opened a new log
I20260621 05:20:33.688601 4411 ts_tablet_manager.cc:1397] T 33955ad97df447d387fb4cbb55698c23 P be1ed1d9dd334995b627a6cddfcf9c30: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:20:33.688773 4411 raft_consensus.cc:348] T 33955ad97df447d387fb4cbb55698c23 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.688851 4412 tablet_bootstrap.cc:492] T 33955ad97df447d387fb4cbb55698c23 P 66e512a22ffa4800926686468233a6c3: No bootstrap required, opened a new log
I20260621 05:20:33.688860 4411 raft_consensus.cc:374] T 33955ad97df447d387fb4cbb55698c23 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:33.688891 4411 raft_consensus.cc:729] T 33955ad97df447d387fb4cbb55698c23 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1ed1d9dd334995b627a6cddfcf9c30, State: Initialized, Role: FOLLOWER
I20260621 05:20:33.688899 4412 ts_tablet_manager.cc:1397] T 33955ad97df447d387fb4cbb55698c23 P 66e512a22ffa4800926686468233a6c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:20:33.688941 4413 tablet_bootstrap.cc:492] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb: Bootstrap starting.
I20260621 05:20:33.688956 4411 consensus_queue.cc:260] T 33955ad97df447d387fb4cbb55698c23 P be1ed1d9dd334995b627a6cddfcf9c30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.689078 4411 ts_tablet_manager.cc:1428] T 33955ad97df447d387fb4cbb55698c23 P be1ed1d9dd334995b627a6cddfcf9c30: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:33.689116 4412 raft_consensus.cc:348] T 33955ad97df447d387fb4cbb55698c23 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.689244 4412 raft_consensus.cc:374] T 33955ad97df447d387fb4cbb55698c23 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:33.689272 4412 raft_consensus.cc:729] T 33955ad97df447d387fb4cbb55698c23 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e512a22ffa4800926686468233a6c3, State: Initialized, Role: FOLLOWER
I20260621 05:20:33.689328 4412 consensus_queue.cc:260] T 33955ad97df447d387fb4cbb55698c23 P 66e512a22ffa4800926686468233a6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.689309 4417 tablet_replica.cc:439] T 33955ad97df447d387fb4cbb55698c23 P be1ed1d9dd334995b627a6cddfcf9c30: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } } }
I20260621 05:20:33.689383 4413 tablet_bootstrap.cc:654] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:33.689404 4417 tablet_replica.cc:442] T 33955ad97df447d387fb4cbb55698c23 P be1ed1d9dd334995b627a6cddfcf9c30: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:20:33.689445 4412 ts_tablet_manager.cc:1428] T 33955ad97df447d387fb4cbb55698c23 P 66e512a22ffa4800926686468233a6c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260621 05:20:33.689514 4418 tablet_replica.cc:439] T 33955ad97df447d387fb4cbb55698c23 P 66e512a22ffa4800926686468233a6c3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } } }
I20260621 05:20:33.689608 4418 tablet_replica.cc:442] T 33955ad97df447d387fb4cbb55698c23 P 66e512a22ffa4800926686468233a6c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:20:33.690371 4413 tablet_bootstrap.cc:492] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb: No bootstrap required, opened a new log
I20260621 05:20:33.690438 4413 ts_tablet_manager.cc:1397] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:20:33.690627 4413 raft_consensus.cc:348] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.690739 4413 raft_consensus.cc:374] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:33.690758 4413 raft_consensus.cc:729] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35c712dea5ca49b4a3da080f62df9adb, State: Initialized, Role: FOLLOWER
I20260621 05:20:33.690815 4413 consensus_queue.cc:260] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.690928 4413 ts_tablet_manager.cc:1428] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:33.690971 4419 tablet_replica.cc:439] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } } }
I20260621 05:20:33.691066 4419 tablet_replica.cc:442] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:20:33.756319 4419 raft_consensus.cc:482] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:20:33.756417 4419 raft_consensus.cc:504] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.756558 4419 leader_election.cc:288] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be1ed1d9dd334995b627a6cddfcf9c30 (127.2.32.1:45881), 66e512a22ffa4800926686468233a6c3 (127.2.32.3:45001)
I20260621 05:20:33.756920 4062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33955ad97df447d387fb4cbb55698c23" candidate_uuid: "35c712dea5ca49b4a3da080f62df9adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" is_pre_election: true
I20260621 05:20:33.757017 4062 raft_consensus.cc:2454] T 33955ad97df447d387fb4cbb55698c23 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35c712dea5ca49b4a3da080f62df9adb in term 0.
I20260621 05:20:33.757160 4136 leader_election.cc:302] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35c712dea5ca49b4a3da080f62df9adb, be1ed1d9dd334995b627a6cddfcf9c30; no voters:
I20260621 05:20:33.757246 4419 raft_consensus.cc:2781] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:20:33.757270 4419 raft_consensus.cc:482] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:20:33.757289 4419 raft_consensus.cc:3037] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:33.758143 4419 raft_consensus.cc:504] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.758262 4419 leader_election.cc:288] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb [CANDIDATE]: Term 1 election: Requested vote from peers be1ed1d9dd334995b627a6cddfcf9c30 (127.2.32.1:45881), 66e512a22ffa4800926686468233a6c3 (127.2.32.3:45001)
I20260621 05:20:33.758621 4341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33955ad97df447d387fb4cbb55698c23" candidate_uuid: "35c712dea5ca49b4a3da080f62df9adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e512a22ffa4800926686468233a6c3"
I20260621 05:20:33.758752 4341 raft_consensus.cc:3037] T 33955ad97df447d387fb4cbb55698c23 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:33.759720 4341 raft_consensus.cc:2454] T 33955ad97df447d387fb4cbb55698c23 P 66e512a22ffa4800926686468233a6c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35c712dea5ca49b4a3da080f62df9adb in term 1.
I20260621 05:20:33.759997 4135 leader_election.cc:302] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35c712dea5ca49b4a3da080f62df9adb, 66e512a22ffa4800926686468233a6c3; no voters:
I20260621 05:20:33.759982 4062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33955ad97df447d387fb4cbb55698c23" candidate_uuid: "35c712dea5ca49b4a3da080f62df9adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1ed1d9dd334995b627a6cddfcf9c30"
I20260621 05:20:33.760046 4062 raft_consensus.cc:3037] T 33955ad97df447d387fb4cbb55698c23 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:33.760066 4419 raft_consensus.cc:2781] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:33.760099 4419 raft_consensus.cc:686] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb [term 1 LEADER]: Becoming Leader. State: Replica: 35c712dea5ca49b4a3da080f62df9adb, State: Running, Role: LEADER
I20260621 05:20:33.760142 4419 consensus_queue.cc:237] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.760437 4419 tablet_replica.cc:439] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb: TxnStatusTablet state changed. Reason: New leader 35c712dea5ca49b4a3da080f62df9adb. Latest consensus state: current_term: 1 leader_uuid: "35c712dea5ca49b4a3da080f62df9adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } } }
I20260621 05:20:33.760504 4419 tablet_replica.cc:442] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:20:33.760753 4062 raft_consensus.cc:2454] T 33955ad97df447d387fb4cbb55698c23 P be1ed1d9dd334995b627a6cddfcf9c30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35c712dea5ca49b4a3da080f62df9adb in term 1.
I20260621 05:20:33.760902 4433 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 05:20:33.761013 4433 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 05:20:33.761059 3905 catalog_manager.cc:5495] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb reported cstate change: term changed from 0 to 1, leader changed from <none> to 35c712dea5ca49b4a3da080f62df9adb (127.2.32.2). New cstate: current_term: 1 leader_uuid: "35c712dea5ca49b4a3da080f62df9adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } health_report { overall_health: UNKNOWN } } }
I20260621 05:20:33.761730 4342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33955ad97df447d387fb4cbb55698c23" candidate_uuid: "35c712dea5ca49b4a3da080f62df9adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e512a22ffa4800926686468233a6c3" is_pre_election: true
I20260621 05:20:33.761828 4342 raft_consensus.cc:2362] T 33955ad97df447d387fb4cbb55698c23 P 66e512a22ffa4800926686468233a6c3 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 35c712dea5ca49b4a3da080f62df9adb in term 1. Re-sending same reply.
I20260621 05:20:33.768666 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.764954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:37907 (local address 127.0.0.1:54792)
0621 05:20:33.765035 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:33.765052 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:33.765118 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:33.765391 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:33.765393 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:33.765394 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:20:33.765447 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.765451 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.766361 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.766362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.767307 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.767312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.768017 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.768019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.768093 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.768111 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:20:33.768474 (+ 363us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:20:33.768482 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 05:20:33.768524 (+ 42us) client_negotiation.cc:240] Negotiation successful
0621 05:20:33.768576 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260621 05:20:33.769008 4410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.765084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54792 (local address 127.2.32.2:37907)
0621 05:20:33.765122 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.765124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.765135 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:20:33.765228 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.765229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.765248 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.765327 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:33.765524 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.766263 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.767437 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.767898 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.768007 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.768258 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:33.768375 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:33.768448 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.768887 (+ 439us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.768917 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:20:33.770891 4342 raft_consensus.cc:1261] T 33955ad97df447d387fb4cbb55698c23 P 66e512a22ffa4800926686468233a6c3 [term 1 FOLLOWER]: Refusing update from remote peer 35c712dea5ca49b4a3da080f62df9adb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:33.770902 4062 raft_consensus.cc:1261] T 33955ad97df447d387fb4cbb55698c23 P be1ed1d9dd334995b627a6cddfcf9c30 [term 1 FOLLOWER]: Refusing update from remote peer 35c712dea5ca49b4a3da080f62df9adb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:33.771370 4421 consensus_queue.cc:1035] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb [LEADER]: Connected to new peer: Peer: permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:33.771744 4419 consensus_queue.cc:1035] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:33.773110 4418 tablet_replica.cc:439] T 33955ad97df447d387fb4cbb55698c23 P 66e512a22ffa4800926686468233a6c3: TxnStatusTablet state changed. Reason: New leader 35c712dea5ca49b4a3da080f62df9adb. Latest consensus state: current_term: 1 leader_uuid: "35c712dea5ca49b4a3da080f62df9adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } } }
I20260621 05:20:33.773223 4418 tablet_replica.cc:442] T 33955ad97df447d387fb4cbb55698c23 P 66e512a22ffa4800926686468233a6c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:20:33.773708 4417 tablet_replica.cc:439] T 33955ad97df447d387fb4cbb55698c23 P be1ed1d9dd334995b627a6cddfcf9c30: TxnStatusTablet state changed. Reason: New leader 35c712dea5ca49b4a3da080f62df9adb. Latest consensus state: current_term: 1 leader_uuid: "35c712dea5ca49b4a3da080f62df9adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } } }
I20260621 05:20:33.773864 4417 tablet_replica.cc:442] T 33955ad97df447d387fb4cbb55698c23 P be1ed1d9dd334995b627a6cddfcf9c30: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:20:33.774463 4419 tablet_replica.cc:439] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "35c712dea5ca49b4a3da080f62df9adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } } }
I20260621 05:20:33.774545 4419 tablet_replica.cc:442] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:20:33.774595 4419 tablet_replica.cc:439] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "35c712dea5ca49b4a3da080f62df9adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } } }
I20260621 05:20:33.774631 4419 tablet_replica.cc:442] T 33955ad97df447d387fb4cbb55698c23 P 35c712dea5ca49b4a3da080f62df9adb: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:20:33.774695 4436 mvcc.cc:205] Tried to move back new op lower bound from 7299150781522616320 to 7299150781481775104. Current Snapshot: MvccSnapshot[applied={T|T < 7299150781522616320}]
I20260621 05:20:33.775082 4417 tablet_replica.cc:439] T 33955ad97df447d387fb4cbb55698c23 P be1ed1d9dd334995b627a6cddfcf9c30: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "35c712dea5ca49b4a3da080f62df9adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } } }
I20260621 05:20:33.775233 4417 tablet_replica.cc:442] T 33955ad97df447d387fb4cbb55698c23 P be1ed1d9dd334995b627a6cddfcf9c30: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:20:33.775238 4418 tablet_replica.cc:439] T 33955ad97df447d387fb4cbb55698c23 P 66e512a22ffa4800926686468233a6c3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "35c712dea5ca49b4a3da080f62df9adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } } }
I20260621 05:20:33.775315 4418 tablet_replica.cc:442] T 33955ad97df447d387fb4cbb55698c23 P 66e512a22ffa4800926686468233a6c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:20:33.775300 4437 mvcc.cc:205] Tried to move back new op lower bound from 7299150781522616320 to 7299150781481775104. Current Snapshot: MvccSnapshot[applied={T|T < 7299150781522616320}]
I20260621 05:20:33.776281 4439 mvcc.cc:205] Tried to move back new op lower bound from 7299150781522616320 to 7299150781481775104. Current Snapshot: MvccSnapshot[applied={T|T < 7299150781522616320 or (T in {7299150781522616320})}]
I20260621 05:20:33.780663 4410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.777415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54800 (local address 127.2.32.2:37907)
0621 05:20:33.777495 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.777497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.777505 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:20:33.777527 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.777527 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.777536 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.777615 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:33.777839 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.778406 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.779585 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.779791 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.779896 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.780285 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:33.780337 (+ 52us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:33.780406 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.780520 (+ 114us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.780556 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 05:20:33.786993 4254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.784077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:37907 (local address 127.2.32.2:55271)
0621 05:20:33.784176 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:33.784188 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:33.784240 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:33.784480 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:33.784481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:33.784483 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:20:33.784587 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.784589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.785401 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.785402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.786069 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.786071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.786485 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.786486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.786551 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.786576 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:20:33.786810 (+ 234us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:20:33.786812 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:20:33.786871 (+ 59us) client_negotiation.cc:240] Negotiation successful
0621 05:20:33.786915 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":2}
I20260621 05:20:33.786993 4410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.784297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:55271 (local address 127.2.32.2:37907)
0621 05:20:33.784363 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.784365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.784372 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:20:33.784394 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.784395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.784405 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.784480 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:33.784654 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.785251 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.786187 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.786395 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.786486 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.786687 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:33.786722 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:33.786807 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.786874 (+ 67us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.786894 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":47}
I20260621 05:20:33.790438 4062 raft_consensus.cc:1261] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30 [term 1 FOLLOWER]: Refusing update from remote peer 35c712dea5ca49b4a3da080f62df9adb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:33.790468 4342 raft_consensus.cc:1261] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3 [term 1 FOLLOWER]: Refusing update from remote peer 35c712dea5ca49b4a3da080f62df9adb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:33.790665 4421 consensus_queue.cc:1035] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [LEADER]: Connected to new peer: Peer: permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:33.790813 4421 consensus_queue.cc:1035] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:33.807487 3904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:20:33.807629 3904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:20:33.808737 3904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f: Sending DeleteTablet for 3 replicas of tablet ac8d0a06d86a42be9a16872a7d599afb
I20260621 05:20:33.809265 4042 tablet_service.cc:1514] Processing DeleteTablet for tablet ac8d0a06d86a42be9a16872a7d599afb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:33 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:47050
I20260621 05:20:33.809317 4322 tablet_service.cc:1514] Processing DeleteTablet for tablet ac8d0a06d86a42be9a16872a7d599afb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:33 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:39226
I20260621 05:20:33.809618 4452 tablet_replica.cc:330] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30: stopping tablet replica
I20260621 05:20:33.809922 4452 raft_consensus.cc:2229] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:20:33.810067 4452 raft_consensus.cc:2258] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:33.810101 4182 tablet_service.cc:1514] Processing DeleteTablet for tablet ac8d0a06d86a42be9a16872a7d599afb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:33 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:54786
I20260621 05:20:33.810196 3904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:20:33.810356 3904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:20:33.810451 4453 tablet_replica.cc:330] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb: stopping tablet replica
I20260621 05:20:33.810585 4451 tablet_replica.cc:330] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3: stopping tablet replica
I20260621 05:20:33.810885 4451 raft_consensus.cc:2229] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:20:33.811000 4451 raft_consensus.cc:2258] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:33.811065 4452 ts_tablet_manager.cc:1905] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:33.811637 4451 ts_tablet_manager.cc:1905] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:33.812420 4182 tablet_service.cc:1467] Processing CreateTablet for tablet 62c626ae84004fe2910153a247b79a64 (DEFAULT_TABLE table=test-table [id=55f521eee9504d458940c649c54ab0e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:33.812551 4182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c626ae84004fe2910153a247b79a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:33.813208 4452 ts_tablet_manager.cc:1918] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:20:33.813468 4452 log.cc:1196] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-0/wal/wals/ac8d0a06d86a42be9a16872a7d599afb
I20260621 05:20:33.813853 4452 ts_tablet_manager.cc:1939] T ac8d0a06d86a42be9a16872a7d599afb P be1ed1d9dd334995b627a6cddfcf9c30: Deleting consensus metadata
I20260621 05:20:33.814178 4451 ts_tablet_manager.cc:1918] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:20:33.814280 4451 log.cc:1196] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/ac8d0a06d86a42be9a16872a7d599afb
I20260621 05:20:33.814538 4451 ts_tablet_manager.cc:1939] T ac8d0a06d86a42be9a16872a7d599afb P 66e512a22ffa4800926686468233a6c3: Deleting consensus metadata
I20260621 05:20:33.815084 4413 tablet_bootstrap.cc:492] T 62c626ae84004fe2910153a247b79a64 P 35c712dea5ca49b4a3da080f62df9adb: Bootstrap starting.
I20260621 05:20:33.815996 4413 tablet_bootstrap.cc:654] T 62c626ae84004fe2910153a247b79a64 P 35c712dea5ca49b4a3da080f62df9adb: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:33.815990 4042 tablet_service.cc:1467] Processing CreateTablet for tablet 62c626ae84004fe2910153a247b79a64 (DEFAULT_TABLE table=test-table [id=55f521eee9504d458940c649c54ab0e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:33.816123 4042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c626ae84004fe2910153a247b79a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:33.816882 4413 tablet_bootstrap.cc:492] T 62c626ae84004fe2910153a247b79a64 P 35c712dea5ca49b4a3da080f62df9adb: No bootstrap required, opened a new log
I20260621 05:20:33.816951 4413 ts_tablet_manager.cc:1397] T 62c626ae84004fe2910153a247b79a64 P 35c712dea5ca49b4a3da080f62df9adb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260621 05:20:33.817147 4413 raft_consensus.cc:348] T 62c626ae84004fe2910153a247b79a64 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.817237 4413 raft_consensus.cc:374] T 62c626ae84004fe2910153a247b79a64 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:33.817257 4413 raft_consensus.cc:729] T 62c626ae84004fe2910153a247b79a64 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35c712dea5ca49b4a3da080f62df9adb, State: Initialized, Role: FOLLOWER
I20260621 05:20:33.817298 4413 consensus_queue.cc:260] T 62c626ae84004fe2910153a247b79a64 P 35c712dea5ca49b4a3da080f62df9adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.817399 4413 ts_tablet_manager.cc:1428] T 62c626ae84004fe2910153a247b79a64 P 35c712dea5ca49b4a3da080f62df9adb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:33.817795 4411 tablet_bootstrap.cc:492] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30: Bootstrap starting.
I20260621 05:20:33.818287 4411 tablet_bootstrap.cc:654] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:33.819099 4411 tablet_bootstrap.cc:492] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30: No bootstrap required, opened a new log
I20260621 05:20:33.819190 4411 ts_tablet_manager.cc:1397] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:33.819537 4411 raft_consensus.cc:348] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.819689 4411 raft_consensus.cc:374] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:33.819722 4411 raft_consensus.cc:729] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1ed1d9dd334995b627a6cddfcf9c30, State: Initialized, Role: FOLLOWER
I20260621 05:20:33.819743 3891 catalog_manager.cc:4841] TS 66e512a22ffa4800926686468233a6c3 (127.2.32.3:45001): tablet ac8d0a06d86a42be9a16872a7d599afb (table test-table [id=d49744b708a9462ba1cb9d80127b3c3d]) successfully deleted
I20260621 05:20:33.819805 4411 consensus_queue.cc:260] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.819912 4453 raft_consensus.cc:2229] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:20:33.819926 4411 ts_tablet_manager.cc:1428] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:33.820072 4453 raft_consensus.cc:2258] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:33.820456 4322 tablet_service.cc:1467] Processing CreateTablet for tablet 62c626ae84004fe2910153a247b79a64 (DEFAULT_TABLE table=test-table [id=55f521eee9504d458940c649c54ab0e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:33.820590 4322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c626ae84004fe2910153a247b79a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:33.820834 4453 ts_tablet_manager.cc:1905] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:33.822167 4412 tablet_bootstrap.cc:492] T 62c626ae84004fe2910153a247b79a64 P 66e512a22ffa4800926686468233a6c3: Bootstrap starting.
I20260621 05:20:33.822573 4412 tablet_bootstrap.cc:654] T 62c626ae84004fe2910153a247b79a64 P 66e512a22ffa4800926686468233a6c3: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:33.822873 3892 catalog_manager.cc:4841] TS be1ed1d9dd334995b627a6cddfcf9c30 (127.2.32.1:45881): tablet ac8d0a06d86a42be9a16872a7d599afb (table test-table [id=d49744b708a9462ba1cb9d80127b3c3d]) successfully deleted
I20260621 05:20:33.823025 4453 ts_tablet_manager.cc:1918] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:20:33.823102 4453 log.cc:1196] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/ac8d0a06d86a42be9a16872a7d599afb
I20260621 05:20:33.823544 4453 ts_tablet_manager.cc:1939] T ac8d0a06d86a42be9a16872a7d599afb P 35c712dea5ca49b4a3da080f62df9adb: Deleting consensus metadata
I20260621 05:20:33.823964 3889 catalog_manager.cc:4841] TS 35c712dea5ca49b4a3da080f62df9adb (127.2.32.2:37907): tablet ac8d0a06d86a42be9a16872a7d599afb (table test-table [id=d49744b708a9462ba1cb9d80127b3c3d]) successfully deleted
I20260621 05:20:33.827781 4412 tablet_bootstrap.cc:492] T 62c626ae84004fe2910153a247b79a64 P 66e512a22ffa4800926686468233a6c3: No bootstrap required, opened a new log
I20260621 05:20:33.827876 4412 ts_tablet_manager.cc:1397] T 62c626ae84004fe2910153a247b79a64 P 66e512a22ffa4800926686468233a6c3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260621 05:20:33.828073 4412 raft_consensus.cc:348] T 62c626ae84004fe2910153a247b79a64 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.828176 4412 raft_consensus.cc:374] T 62c626ae84004fe2910153a247b79a64 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:33.828200 4412 raft_consensus.cc:729] T 62c626ae84004fe2910153a247b79a64 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e512a22ffa4800926686468233a6c3, State: Initialized, Role: FOLLOWER
I20260621 05:20:33.828251 4412 consensus_queue.cc:260] T 62c626ae84004fe2910153a247b79a64 P 66e512a22ffa4800926686468233a6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.828373 4412 ts_tablet_manager.cc:1428] T 62c626ae84004fe2910153a247b79a64 P 66e512a22ffa4800926686468233a6c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260621 05:20:33.853996 4417 raft_consensus.cc:482] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:20:33.854127 4417 raft_consensus.cc:504] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.854466 4417 leader_election.cc:288] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35c712dea5ca49b4a3da080f62df9adb (127.2.32.2:37907), 66e512a22ffa4800926686468233a6c3 (127.2.32.3:45001)
I20260621 05:20:33.858855 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.854850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:56653 (local address 127.2.32.3:45001)
0621 05:20:33.854944 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.854948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.855064 (+ 116us) server_negotiation.cc:407] Connection header received
0621 05:20:33.855203 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.855204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.855215 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.855245 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:33.855526 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.856083 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.857599 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.858170 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.858353 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.858370 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.858694 (+ 324us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.858766 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 05:20:33.858855 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.854696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:45001 (local address 127.2.32.1:56653)
0621 05:20:33.854991 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:33.855004 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:33.855094 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:33.855327 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:33.855328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:33.855331 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:33.855410 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.855412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.856184 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.856185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.857490 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.857495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.858363 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.858365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.858471 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.858475 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:20:33.858682 (+ 207us) client_negotiation.cc:240] Negotiation successful
0621 05:20:33.858754 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":66,"threads_started":1}
I20260621 05:20:33.859362 4342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c626ae84004fe2910153a247b79a64" candidate_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e512a22ffa4800926686468233a6c3" is_pre_election: true
I20260621 05:20:33.859464 4342 raft_consensus.cc:2454] T 62c626ae84004fe2910153a247b79a64 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be1ed1d9dd334995b627a6cddfcf9c30 in term 0.
I20260621 05:20:33.859512 4410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.855011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:57005 (local address 127.2.32.2:37907)
0621 05:20:33.855207 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.855213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.855418 (+ 205us) server_negotiation.cc:407] Connection header received
0621 05:20:33.856254 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.856256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.856264 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.856331 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:33.856537 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.857004 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.858188 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.858894 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.858984 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.859001 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.859400 (+ 399us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.859446 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260621 05:20:33.859530 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.854854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:37907 (local address 127.2.32.1:57005)
0621 05:20:33.855267 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:33.855279 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:33.855447 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:33.856361 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:33.856362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:33.856365 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:33.856466 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.856467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.857090 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.857091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.858077 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.858079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.859122 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.859123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.859292 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.859296 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:20:33.859383 (+ 87us) client_negotiation.cc:240] Negotiation successful
0621 05:20:33.859431 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":149,"threads_started":1}
I20260621 05:20:33.859735 3995 leader_election.cc:302] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e512a22ffa4800926686468233a6c3, be1ed1d9dd334995b627a6cddfcf9c30; no voters:
I20260621 05:20:33.859869 4417 raft_consensus.cc:2781] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:20:33.859920 4417 raft_consensus.cc:482] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:20:33.859946 4417 raft_consensus.cc:3037] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:33.859992 4202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c626ae84004fe2910153a247b79a64" candidate_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35c712dea5ca49b4a3da080f62df9adb" is_pre_election: true
I20260621 05:20:33.860116 4202 raft_consensus.cc:2454] T 62c626ae84004fe2910153a247b79a64 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be1ed1d9dd334995b627a6cddfcf9c30 in term 0.
I20260621 05:20:33.860764 4417 raft_consensus.cc:504] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.860909 4417 leader_election.cc:288] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [CANDIDATE]: Term 1 election: Requested vote from peers 35c712dea5ca49b4a3da080f62df9adb (127.2.32.2:37907), 66e512a22ffa4800926686468233a6c3 (127.2.32.3:45001)
I20260621 05:20:33.861043 4342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c626ae84004fe2910153a247b79a64" candidate_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e512a22ffa4800926686468233a6c3"
I20260621 05:20:33.861055 4202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c626ae84004fe2910153a247b79a64" candidate_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35c712dea5ca49b4a3da080f62df9adb"
I20260621 05:20:33.861130 4202 raft_consensus.cc:3037] T 62c626ae84004fe2910153a247b79a64 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:33.861130 4342 raft_consensus.cc:3037] T 62c626ae84004fe2910153a247b79a64 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:33.861730 4342 raft_consensus.cc:2454] T 62c626ae84004fe2910153a247b79a64 P 66e512a22ffa4800926686468233a6c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be1ed1d9dd334995b627a6cddfcf9c30 in term 1.
I20260621 05:20:33.861887 3995 leader_election.cc:302] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e512a22ffa4800926686468233a6c3, be1ed1d9dd334995b627a6cddfcf9c30; no voters:
I20260621 05:20:33.861928 4202 raft_consensus.cc:2454] T 62c626ae84004fe2910153a247b79a64 P 35c712dea5ca49b4a3da080f62df9adb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be1ed1d9dd334995b627a6cddfcf9c30 in term 1.
I20260621 05:20:33.861966 4417 raft_consensus.cc:2781] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:33.862111 4417 raft_consensus.cc:686] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [term 1 LEADER]: Becoming Leader. State: Replica: be1ed1d9dd334995b627a6cddfcf9c30, State: Running, Role: LEADER
I20260621 05:20:33.862185 4417 consensus_queue.cc:237] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:33.862883 3904 catalog_manager.cc:5495] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 reported cstate change: term changed from 0 to 1, leader changed from <none> to be1ed1d9dd334995b627a6cddfcf9c30 (127.2.32.1). New cstate: current_term: 1 leader_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } health_report { overall_health: UNKNOWN } } }
I20260621 05:20:33.879331 4408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.875978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47062 (local address 127.2.32.1:45881)
0621 05:20:33.876024 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.876026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.876097 (+ 71us) server_negotiation.cc:407] Connection header received
0621 05:20:33.876193 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.876194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.876215 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.876297 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:33.876436 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.877232 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.878160 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.878391 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.878490 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.878666 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:33.878871 (+ 205us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:33.879124 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.879198 (+ 74us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.879242 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:20:33.884588 4410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.881816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:38607 (local address 127.2.32.2:37907)
0621 05:20:33.881856 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.881858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.881990 (+ 132us) server_negotiation.cc:407] Connection header received
0621 05:20:33.882054 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.882055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.882067 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.882138 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:33.882312 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.882990 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.883864 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.884068 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.884156 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.884330 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:33.884369 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:33.884432 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.884498 (+ 66us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.884516 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:20:33.884614 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.881740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:37907 (local address 127.2.32.1:38607)
0621 05:20:33.881924 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:33.881933 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:33.882005 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:33.882126 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:33.882127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:33.882129 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:20:33.882209 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.882210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.883080 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.883081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.883763 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.883765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.884157 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.884158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.884224 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.884237 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:20:33.884432 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:20:33.884434 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:20:33.884492 (+ 58us) client_negotiation.cc:240] Negotiation successful
0621 05:20:33.884530 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":94,"threads_started":1}
I20260621 05:20:33.887014 4342 raft_consensus.cc:1261] T 62c626ae84004fe2910153a247b79a64 P 66e512a22ffa4800926686468233a6c3 [term 1 FOLLOWER]: Refusing update from remote peer be1ed1d9dd334995b627a6cddfcf9c30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:33.887104 4202 raft_consensus.cc:1261] T 62c626ae84004fe2910153a247b79a64 P 35c712dea5ca49b4a3da080f62df9adb [term 1 FOLLOWER]: Refusing update from remote peer be1ed1d9dd334995b627a6cddfcf9c30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:33.887383 4417 consensus_queue.cc:1035] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:33.887526 4459 consensus_queue.cc:1035] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:33.895951 4408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.893304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:58247 (local address 127.2.32.1:45881)
0621 05:20:33.893349 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:33.893352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:33.893362 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:20:33.893388 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:33.893389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:33.893405 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:33.893470 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:33.893732 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.894298 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.895112 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.895434 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.895530 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.895691 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:33.895724 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:33.895782 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:33.895857 (+ 75us) server_negotiation.cc:299] Negotiation successful
0621 05:20:33.895879 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:20:33.895963 4254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:33.893017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:45881 (local address 127.2.32.2:58247)
0621 05:20:33.893059 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:33.893069 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:33.893104 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:33.893567 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:33.893572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:33.893574 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:20:33.893626 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.893629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.894407 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.894408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.895014 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:33.895016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:33.895532 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:33.895533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:33.895597 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:33.895601 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:20:33.895794 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:20:33.895796 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:20:33.895852 (+ 56us) client_negotiation.cc:240] Negotiation successful
0621 05:20:33.895893 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
W20260621 05:20:33.896347 4439 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260621 05:20:35.423519 3904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:20:35.423671 3904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:20:35.425324 3904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f: Sending DeleteTablet for 3 replicas of tablet 62c626ae84004fe2910153a247b79a64
I20260621 05:20:35.425806 4182 tablet_service.cc:1514] Processing DeleteTablet for tablet 62c626ae84004fe2910153a247b79a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:35 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:54786
I20260621 05:20:35.425782 4322 tablet_service.cc:1514] Processing DeleteTablet for tablet 62c626ae84004fe2910153a247b79a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:35 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:39226
I20260621 05:20:35.426066 4042 tablet_service.cc:1514] Processing DeleteTablet for tablet 62c626ae84004fe2910153a247b79a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:35 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:47050
I20260621 05:20:35.426115 4490 tablet_replica.cc:330] T 62c626ae84004fe2910153a247b79a64 P 35c712dea5ca49b4a3da080f62df9adb: stopping tablet replica
I20260621 05:20:35.426206 4490 raft_consensus.cc:2229] T 62c626ae84004fe2910153a247b79a64 P 35c712dea5ca49b4a3da080f62df9adb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:20:35.426275 4490 raft_consensus.cc:2258] T 62c626ae84004fe2910153a247b79a64 P 35c712dea5ca49b4a3da080f62df9adb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:35.426402 4492 tablet_replica.cc:330] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30: stopping tablet replica
I20260621 05:20:35.426512 4492 raft_consensus.cc:2229] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:20:35.426630 4490 ts_tablet_manager.cc:1905] T 62c626ae84004fe2910153a247b79a64 P 35c712dea5ca49b4a3da080f62df9adb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:35.426671 4492 raft_consensus.cc:2258] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:35.429149 4490 ts_tablet_manager.cc:1918] T 62c626ae84004fe2910153a247b79a64 P 35c712dea5ca49b4a3da080f62df9adb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 05:20:35.429355 4490 log.cc:1196] T 62c626ae84004fe2910153a247b79a64 P 35c712dea5ca49b4a3da080f62df9adb: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/62c626ae84004fe2910153a247b79a64
I20260621 05:20:35.429663 4490 ts_tablet_manager.cc:1939] T 62c626ae84004fe2910153a247b79a64 P 35c712dea5ca49b4a3da080f62df9adb: Deleting consensus metadata
I20260621 05:20:35.430102 3889 catalog_manager.cc:4841] TS 35c712dea5ca49b4a3da080f62df9adb (127.2.32.2:37907): tablet 62c626ae84004fe2910153a247b79a64 (table test-table [id=55f521eee9504d458940c649c54ab0e9]) successfully deleted
I20260621 05:20:35.430363 4492 ts_tablet_manager.cc:1905] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:35.432206 4492 ts_tablet_manager.cc:1918] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 05:20:35.432281 4492 log.cc:1196] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-0/wal/wals/62c626ae84004fe2910153a247b79a64
I20260621 05:20:35.432570 4491 tablet_replica.cc:330] T 62c626ae84004fe2910153a247b79a64 P 66e512a22ffa4800926686468233a6c3: stopping tablet replica
I20260621 05:20:35.432612 4492 ts_tablet_manager.cc:1939] T 62c626ae84004fe2910153a247b79a64 P be1ed1d9dd334995b627a6cddfcf9c30: Deleting consensus metadata
I20260621 05:20:35.432654 4491 raft_consensus.cc:2229] T 62c626ae84004fe2910153a247b79a64 P 66e512a22ffa4800926686468233a6c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:20:35.432700 4491 raft_consensus.cc:2258] T 62c626ae84004fe2910153a247b79a64 P 66e512a22ffa4800926686468233a6c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:35.432968 3892 catalog_manager.cc:4841] TS be1ed1d9dd334995b627a6cddfcf9c30 (127.2.32.1:45881): tablet 62c626ae84004fe2910153a247b79a64 (table test-table [id=55f521eee9504d458940c649c54ab0e9]) successfully deleted
I20260621 05:20:35.433188 4491 ts_tablet_manager.cc:1905] T 62c626ae84004fe2910153a247b79a64 P 66e512a22ffa4800926686468233a6c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:35.434924 4491 ts_tablet_manager.cc:1918] T 62c626ae84004fe2910153a247b79a64 P 66e512a22ffa4800926686468233a6c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 05:20:35.435019 4491 log.cc:1196] T 62c626ae84004fe2910153a247b79a64 P 66e512a22ffa4800926686468233a6c3: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/62c626ae84004fe2910153a247b79a64
I20260621 05:20:35.435294 4491 ts_tablet_manager.cc:1939] T 62c626ae84004fe2910153a247b79a64 P 66e512a22ffa4800926686468233a6c3: Deleting consensus metadata
I20260621 05:20:35.435657 3891 catalog_manager.cc:4841] TS 66e512a22ffa4800926686468233a6c3 (127.2.32.3:45001): tablet 62c626ae84004fe2910153a247b79a64 (table test-table [id=55f521eee9504d458940c649c54ab0e9]) successfully deleted
Jun 21 05:20:35 dist-test-slave-b357 krb5kdc[3860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019235, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 05:20:35 dist-test-slave-b357 krb5kdc[3860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019235, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:35.457391 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:35.446878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52758 (local address 127.2.32.62:45741)
0621 05:20:35.447112 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:35.447114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:35.447232 (+ 118us) server_negotiation.cc:407] Connection header received
0621 05:20:35.447272 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:35.447273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:35.447289 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:35.447411 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:35.448436 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.449164 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.450252 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.450622 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.450721 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:35.453747 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:35.453749 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:35.453749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:35.453759 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:35.455830 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:35.456261 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:35.456263 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:35.456263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:35.456337 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:35.456662 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:35.456663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:35.456664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:35.456899 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:35.457065 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:35.457205 (+ 140us) server_negotiation.cc:299] Negotiation successful
0621 05:20:35.457313 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":135,"threads_started":1}
I20260621 05:20:35.459228 3904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:20:35.459394 3904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:20:35.460934 4322 tablet_service.cc:1467] Processing CreateTablet for tablet 66374d04e81e499b8c7f621e1bcd0f67 (DEFAULT_TABLE table=test-table [id=ecb4d293a37d4067b0d8170e9912b785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:35.460933 4042 tablet_service.cc:1467] Processing CreateTablet for tablet 66374d04e81e499b8c7f621e1bcd0f67 (DEFAULT_TABLE table=test-table [id=ecb4d293a37d4067b0d8170e9912b785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:35.461032 4182 tablet_service.cc:1467] Processing CreateTablet for tablet 66374d04e81e499b8c7f621e1bcd0f67 (DEFAULT_TABLE table=test-table [id=ecb4d293a37d4067b0d8170e9912b785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:35.461076 4322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66374d04e81e499b8c7f621e1bcd0f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:35.461077 4042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66374d04e81e499b8c7f621e1bcd0f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:35.461164 4182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66374d04e81e499b8c7f621e1bcd0f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:35.462479 4503 tablet_bootstrap.cc:492] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3: Bootstrap starting.
I20260621 05:20:35.462612 4501 tablet_bootstrap.cc:492] T 66374d04e81e499b8c7f621e1bcd0f67 P 35c712dea5ca49b4a3da080f62df9adb: Bootstrap starting.
I20260621 05:20:35.462884 4503 tablet_bootstrap.cc:654] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:35.463008 4501 tablet_bootstrap.cc:654] T 66374d04e81e499b8c7f621e1bcd0f67 P 35c712dea5ca49b4a3da080f62df9adb: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:35.463222 4502 tablet_bootstrap.cc:492] T 66374d04e81e499b8c7f621e1bcd0f67 P be1ed1d9dd334995b627a6cddfcf9c30: Bootstrap starting.
I20260621 05:20:35.463580 4503 tablet_bootstrap.cc:492] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3: No bootstrap required, opened a new log
I20260621 05:20:35.463636 4503 ts_tablet_manager.cc:1397] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:35.463676 4502 tablet_bootstrap.cc:654] T 66374d04e81e499b8c7f621e1bcd0f67 P be1ed1d9dd334995b627a6cddfcf9c30: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:35.463747 4501 tablet_bootstrap.cc:492] T 66374d04e81e499b8c7f621e1bcd0f67 P 35c712dea5ca49b4a3da080f62df9adb: No bootstrap required, opened a new log
I20260621 05:20:35.463799 4501 ts_tablet_manager.cc:1397] T 66374d04e81e499b8c7f621e1bcd0f67 P 35c712dea5ca49b4a3da080f62df9adb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:35.463778 4503 raft_consensus.cc:348] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:35.463860 4503 raft_consensus.cc:374] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:35.463887 4503 raft_consensus.cc:729] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e512a22ffa4800926686468233a6c3, State: Initialized, Role: FOLLOWER
I20260621 05:20:35.463929 4501 raft_consensus.cc:348] T 66374d04e81e499b8c7f621e1bcd0f67 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:35.463997 4501 raft_consensus.cc:374] T 66374d04e81e499b8c7f621e1bcd0f67 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:35.464025 4501 raft_consensus.cc:729] T 66374d04e81e499b8c7f621e1bcd0f67 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35c712dea5ca49b4a3da080f62df9adb, State: Initialized, Role: FOLLOWER
I20260621 05:20:35.463977 4503 consensus_queue.cc:260] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:35.464092 4501 consensus_queue.cc:260] T 66374d04e81e499b8c7f621e1bcd0f67 P 35c712dea5ca49b4a3da080f62df9adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:35.464164 4503 ts_tablet_manager.cc:1428] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260621 05:20:35.464231 4501 ts_tablet_manager.cc:1428] T 66374d04e81e499b8c7f621e1bcd0f67 P 35c712dea5ca49b4a3da080f62df9adb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:20:35.464275 4502 tablet_bootstrap.cc:492] T 66374d04e81e499b8c7f621e1bcd0f67 P be1ed1d9dd334995b627a6cddfcf9c30: No bootstrap required, opened a new log
I20260621 05:20:35.464318 4502 ts_tablet_manager.cc:1397] T 66374d04e81e499b8c7f621e1bcd0f67 P be1ed1d9dd334995b627a6cddfcf9c30: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:35.464434 4502 raft_consensus.cc:348] T 66374d04e81e499b8c7f621e1bcd0f67 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:35.464525 4502 raft_consensus.cc:374] T 66374d04e81e499b8c7f621e1bcd0f67 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:35.464547 4502 raft_consensus.cc:729] T 66374d04e81e499b8c7f621e1bcd0f67 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1ed1d9dd334995b627a6cddfcf9c30, State: Initialized, Role: FOLLOWER
I20260621 05:20:35.464591 4502 consensus_queue.cc:260] T 66374d04e81e499b8c7f621e1bcd0f67 P be1ed1d9dd334995b627a6cddfcf9c30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:35.464681 4502 ts_tablet_manager.cc:1428] T 66374d04e81e499b8c7f621e1bcd0f67 P be1ed1d9dd334995b627a6cddfcf9c30: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:35.514485 4507 raft_consensus.cc:482] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:20:35.514581 4507 raft_consensus.cc:504] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:35.515203 4507 leader_election.cc:288] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be1ed1d9dd334995b627a6cddfcf9c30 (127.2.32.1:45881), 35c712dea5ca49b4a3da080f62df9adb (127.2.32.2:37907)
I20260621 05:20:35.519714 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:35.515503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:33523 (local address 127.2.32.1:45881)
0621 05:20:35.515909 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:35.515911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:35.515925 (+ 14us) server_negotiation.cc:407] Connection header received
0621 05:20:35.515955 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:35.515956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:35.515969 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:35.516077 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:35.516429 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.517319 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.518506 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.519112 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.519213 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:35.519236 (+ 23us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:35.519588 (+ 352us) server_negotiation.cc:299] Negotiation successful
0621 05:20:35.519641 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":269,"threads_started":1}
I20260621 05:20:35.519714 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:35.515503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:56769 (local address 127.2.32.2:37907)
0621 05:20:35.515908 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:35.515911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:35.515925 (+ 14us) server_negotiation.cc:407] Connection header received
0621 05:20:35.515956 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:35.515956 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:35.515969 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:35.516077 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:35.516429 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.517399 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.518597 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.519217 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.519378 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:35.519394 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:35.519585 (+ 191us) server_negotiation.cc:299] Negotiation successful
0621 05:20:35.519634 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":271,"threads_started":1}
I20260621 05:20:35.519702 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:35.515405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:37907 (local address 127.2.32.3:56769)
0621 05:20:35.515683 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:35.515698 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:35.515776 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:35.516139 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:35.516141 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:35.516145 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:35.516282 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:35.516285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.517504 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.517505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:35.518456 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:35.518458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.519375 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.519376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:35.519435 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:35.519440 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:20:35.519506 (+ 66us) client_negotiation.cc:240] Negotiation successful
0621 05:20:35.519558 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":89,"thread_start_us":191,"threads_started":1}
I20260621 05:20:35.519701 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:35.515414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:45881 (local address 127.2.32.3:33523)
0621 05:20:35.515683 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:35.515704 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:35.515776 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:35.516151 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:35.516151 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:35.516153 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:35.516282 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:35.516285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.517449 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.517451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:35.518416 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:35.518419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.519352 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.519353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:35.519433 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:35.519438 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:20:35.519506 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 05:20:35.519558 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":9,"spinlock_wait_cycles":1280,"thread_start_us":187,"threads_started":1}
I20260621 05:20:35.520090 4062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66374d04e81e499b8c7f621e1bcd0f67" candidate_uuid: "66e512a22ffa4800926686468233a6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" is_pre_election: true
I20260621 05:20:35.520144 4202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66374d04e81e499b8c7f621e1bcd0f67" candidate_uuid: "66e512a22ffa4800926686468233a6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35c712dea5ca49b4a3da080f62df9adb" is_pre_election: true
I20260621 05:20:35.520200 4062 raft_consensus.cc:2454] T 66374d04e81e499b8c7f621e1bcd0f67 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e512a22ffa4800926686468233a6c3 in term 0.
I20260621 05:20:35.520216 4202 raft_consensus.cc:2454] T 66374d04e81e499b8c7f621e1bcd0f67 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e512a22ffa4800926686468233a6c3 in term 0.
I20260621 05:20:35.520430 4273 leader_election.cc:302] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35c712dea5ca49b4a3da080f62df9adb, 66e512a22ffa4800926686468233a6c3; no voters:
I20260621 05:20:35.520617 4507 raft_consensus.cc:2781] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:20:35.520684 4507 raft_consensus.cc:482] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:20:35.520712 4507 raft_consensus.cc:3037] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:35.521466 4507 raft_consensus.cc:504] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:35.521638 4507 leader_election.cc:288] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [CANDIDATE]: Term 1 election: Requested vote from peers be1ed1d9dd334995b627a6cddfcf9c30 (127.2.32.1:45881), 35c712dea5ca49b4a3da080f62df9adb (127.2.32.2:37907)
I20260621 05:20:35.521899 4062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66374d04e81e499b8c7f621e1bcd0f67" candidate_uuid: "66e512a22ffa4800926686468233a6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1ed1d9dd334995b627a6cddfcf9c30"
I20260621 05:20:35.521921 4202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66374d04e81e499b8c7f621e1bcd0f67" candidate_uuid: "66e512a22ffa4800926686468233a6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35c712dea5ca49b4a3da080f62df9adb"
I20260621 05:20:35.521978 4202 raft_consensus.cc:3037] T 66374d04e81e499b8c7f621e1bcd0f67 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:35.521978 4062 raft_consensus.cc:3037] T 66374d04e81e499b8c7f621e1bcd0f67 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:35.522745 4202 raft_consensus.cc:2454] T 66374d04e81e499b8c7f621e1bcd0f67 P 35c712dea5ca49b4a3da080f62df9adb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e512a22ffa4800926686468233a6c3 in term 1.
I20260621 05:20:35.522745 4062 raft_consensus.cc:2454] T 66374d04e81e499b8c7f621e1bcd0f67 P be1ed1d9dd334995b627a6cddfcf9c30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e512a22ffa4800926686468233a6c3 in term 1.
I20260621 05:20:35.522948 4273 leader_election.cc:302] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35c712dea5ca49b4a3da080f62df9adb, 66e512a22ffa4800926686468233a6c3; no voters:
I20260621 05:20:35.523051 4507 raft_consensus.cc:2781] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:35.523231 4507 raft_consensus.cc:686] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [term 1 LEADER]: Becoming Leader. State: Replica: 66e512a22ffa4800926686468233a6c3, State: Running, Role: LEADER
I20260621 05:20:35.523320 4507 consensus_queue.cc:237] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:35.524430 3903 catalog_manager.cc:5495] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e512a22ffa4800926686468233a6c3 (127.2.32.3). New cstate: current_term: 1 leader_uuid: "66e512a22ffa4800926686468233a6c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } health_report { overall_health: HEALTHY } } }
W20260621 05:20:35.545233 4389 tablet.cc:2369] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:20:35.545436 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:35.542671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39230 (local address 127.2.32.3:45001)
0621 05:20:35.542921 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:35.542923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:35.542931 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:20:35.542969 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:35.542969 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:35.542989 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:35.543104 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:35.543343 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.543850 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.544641 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.544852 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.544927 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:35.545099 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:35.545206 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:35.545290 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:35.545331 (+ 41us) server_negotiation.cc:299] Negotiation successful
0621 05:20:35.545351 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":183,"threads_started":1}
I20260621 05:20:35.550977 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:35.548013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:37907 (local address 127.2.32.3:41705)
0621 05:20:35.548256 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:35.548268 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:35.548332 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:35.548521 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:35.548522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:35.548525 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:20:35.548589 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:35.548591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.549332 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.549333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:35.549915 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:35.549917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.550420 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.550421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:35.550466 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:35.550478 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:20:35.550793 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:20:35.550796 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:20:35.550848 (+ 52us) client_negotiation.cc:240] Negotiation successful
0621 05:20:35.550887 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":84,"threads_started":1}
I20260621 05:20:35.550994 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:35.548157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:41705 (local address 127.2.32.2:37907)
0621 05:20:35.548199 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:35.548201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:35.548366 (+ 165us) server_negotiation.cc:407] Connection header received
0621 05:20:35.548387 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:35.548388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:35.548397 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:35.548444 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:35.548689 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.549127 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.550087 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.550310 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.550377 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:35.550659 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:35.550699 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:35.550760 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:35.550861 (+ 101us) server_negotiation.cc:299] Negotiation successful
0621 05:20:35.550893 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:20:35.554384 4202 raft_consensus.cc:1261] T 66374d04e81e499b8c7f621e1bcd0f67 P 35c712dea5ca49b4a3da080f62df9adb [term 1 FOLLOWER]: Refusing update from remote peer 66e512a22ffa4800926686468233a6c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:35.554384 4062 raft_consensus.cc:1261] T 66374d04e81e499b8c7f621e1bcd0f67 P be1ed1d9dd334995b627a6cddfcf9c30 [term 1 FOLLOWER]: Refusing update from remote peer 66e512a22ffa4800926686468233a6c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:35.554646 4512 consensus_queue.cc:1035] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:35.554973 4507 consensus_queue.cc:1035] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:35.562507 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:35.559836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:42311 (local address 127.2.32.3:45001)
0621 05:20:35.559877 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:35.559879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:35.559888 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:20:35.559942 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:35.559943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:35.559953 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:35.560018 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:35.560333 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.560807 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.561786 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.561996 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.562086 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:35.562229 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:35.562259 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:35.562306 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:35.562438 (+ 132us) server_negotiation.cc:299] Negotiation successful
0621 05:20:35.562456 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:20:35.562736 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:35.559636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:45001 (local address 127.2.32.2:42311)
0621 05:20:35.559776 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:35.559787 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:35.559836 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:35.560113 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:35.560115 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:35.560118 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:20:35.560209 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:35.560211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.560920 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.560922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:35.561689 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:35.561691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.562087 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.562088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:35.562142 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:35.562147 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:20:35.562365 (+ 218us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:20:35.562369 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:20:35.562436 (+ 67us) client_negotiation.cc:240] Negotiation successful
0621 05:20:35.562508 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":101,"threads_started":1}
I20260621 05:20:35.573134 3903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:20:35.573280 3903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:20:35.573985 3903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f: Sending DeleteTablet for 3 replicas of tablet 66374d04e81e499b8c7f621e1bcd0f67
I20260621 05:20:35.574209 4042 tablet_service.cc:1514] Processing DeleteTablet for tablet 66374d04e81e499b8c7f621e1bcd0f67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:35 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:47050
I20260621 05:20:35.574251 4322 tablet_service.cc:1514] Processing DeleteTablet for tablet 66374d04e81e499b8c7f621e1bcd0f67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:35 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:39226
I20260621 05:20:35.574281 4182 tablet_service.cc:1514] Processing DeleteTablet for tablet 66374d04e81e499b8c7f621e1bcd0f67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:35 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:54786
I20260621 05:20:35.574311 4492 tablet_replica.cc:330] T 66374d04e81e499b8c7f621e1bcd0f67 P be1ed1d9dd334995b627a6cddfcf9c30: stopping tablet replica
I20260621 05:20:35.574345 4491 tablet_replica.cc:330] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3: stopping tablet replica
I20260621 05:20:35.574362 4492 raft_consensus.cc:2229] T 66374d04e81e499b8c7f621e1bcd0f67 P be1ed1d9dd334995b627a6cddfcf9c30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:20:35.574389 4491 raft_consensus.cc:2229] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:20:35.574414 4492 raft_consensus.cc:2258] T 66374d04e81e499b8c7f621e1bcd0f67 P be1ed1d9dd334995b627a6cddfcf9c30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:35.574476 4491 raft_consensus.cc:2258] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:35.574556 4490 tablet_replica.cc:330] T 66374d04e81e499b8c7f621e1bcd0f67 P 35c712dea5ca49b4a3da080f62df9adb: stopping tablet replica
I20260621 05:20:35.574633 4490 raft_consensus.cc:2229] T 66374d04e81e499b8c7f621e1bcd0f67 P 35c712dea5ca49b4a3da080f62df9adb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:20:35.574683 4490 raft_consensus.cc:2258] T 66374d04e81e499b8c7f621e1bcd0f67 P 35c712dea5ca49b4a3da080f62df9adb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:35.574705 3903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:20:35.574844 3903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:20:35.575004 4491 ts_tablet_manager.cc:1905] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:35.575371 4490 ts_tablet_manager.cc:1905] T 66374d04e81e499b8c7f621e1bcd0f67 P 35c712dea5ca49b4a3da080f62df9adb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:35.575927 4042 tablet_service.cc:1467] Processing CreateTablet for tablet dfc06be6739e4d7ebd518cca130d5f98 (DEFAULT_TABLE table=test-table [id=f3d5f8f043d34df583ff793826fea12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:35.576030 4042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc06be6739e4d7ebd518cca130d5f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:35.576670 4491 ts_tablet_manager.cc:1918] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:20:35.576735 4491 log.cc:1196] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/66374d04e81e499b8c7f621e1bcd0f67
I20260621 05:20:35.576954 4491 ts_tablet_manager.cc:1939] T 66374d04e81e499b8c7f621e1bcd0f67 P 66e512a22ffa4800926686468233a6c3: Deleting consensus metadata
I20260621 05:20:35.576957 4490 ts_tablet_manager.cc:1918] T 66374d04e81e499b8c7f621e1bcd0f67 P 35c712dea5ca49b4a3da080f62df9adb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:20:35.577008 4490 log.cc:1196] T 66374d04e81e499b8c7f621e1bcd0f67 P 35c712dea5ca49b4a3da080f62df9adb: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/66374d04e81e499b8c7f621e1bcd0f67
I20260621 05:20:35.577244 4490 ts_tablet_manager.cc:1939] T 66374d04e81e499b8c7f621e1bcd0f67 P 35c712dea5ca49b4a3da080f62df9adb: Deleting consensus metadata
I20260621 05:20:35.577358 4492 ts_tablet_manager.cc:1905] T 66374d04e81e499b8c7f621e1bcd0f67 P be1ed1d9dd334995b627a6cddfcf9c30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:35.577378 4502 tablet_bootstrap.cc:492] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30: Bootstrap starting.
I20260621 05:20:35.577589 3891 catalog_manager.cc:4841] TS 66e512a22ffa4800926686468233a6c3 (127.2.32.3:45001): tablet 66374d04e81e499b8c7f621e1bcd0f67 (table test-table [id=ecb4d293a37d4067b0d8170e9912b785]) successfully deleted
I20260621 05:20:35.577709 4322 tablet_service.cc:1467] Processing CreateTablet for tablet dfc06be6739e4d7ebd518cca130d5f98 (DEFAULT_TABLE table=test-table [id=f3d5f8f043d34df583ff793826fea12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:35.577749 4502 tablet_bootstrap.cc:654] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:35.577800 4322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc06be6739e4d7ebd518cca130d5f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:35.578665 4503 tablet_bootstrap.cc:492] T dfc06be6739e4d7ebd518cca130d5f98 P 66e512a22ffa4800926686468233a6c3: Bootstrap starting.
I20260621 05:20:35.579002 4182 tablet_service.cc:1467] Processing CreateTablet for tablet dfc06be6739e4d7ebd518cca130d5f98 (DEFAULT_TABLE table=test-table [id=f3d5f8f043d34df583ff793826fea12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:35.579082 4503 tablet_bootstrap.cc:654] T dfc06be6739e4d7ebd518cca130d5f98 P 66e512a22ffa4800926686468233a6c3: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:35.579094 4182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc06be6739e4d7ebd518cca130d5f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:35.579288 4492 ts_tablet_manager.cc:1918] T 66374d04e81e499b8c7f621e1bcd0f67 P be1ed1d9dd334995b627a6cddfcf9c30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:20:35.579344 4492 log.cc:1196] T 66374d04e81e499b8c7f621e1bcd0f67 P be1ed1d9dd334995b627a6cddfcf9c30: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-0/wal/wals/66374d04e81e499b8c7f621e1bcd0f67
I20260621 05:20:35.579552 4492 ts_tablet_manager.cc:1939] T 66374d04e81e499b8c7f621e1bcd0f67 P be1ed1d9dd334995b627a6cddfcf9c30: Deleting consensus metadata
I20260621 05:20:35.579862 4503 tablet_bootstrap.cc:492] T dfc06be6739e4d7ebd518cca130d5f98 P 66e512a22ffa4800926686468233a6c3: No bootstrap required, opened a new log
I20260621 05:20:35.579879 3892 catalog_manager.cc:4841] TS be1ed1d9dd334995b627a6cddfcf9c30 (127.2.32.1:45881): tablet 66374d04e81e499b8c7f621e1bcd0f67 (table test-table [id=ecb4d293a37d4067b0d8170e9912b785]) successfully deleted
I20260621 05:20:35.579913 4503 ts_tablet_manager.cc:1397] T dfc06be6739e4d7ebd518cca130d5f98 P 66e512a22ffa4800926686468233a6c3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:35.580070 4503 raft_consensus.cc:348] T dfc06be6739e4d7ebd518cca130d5f98 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:35.580127 4503 raft_consensus.cc:374] T dfc06be6739e4d7ebd518cca130d5f98 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:35.580147 4503 raft_consensus.cc:729] T dfc06be6739e4d7ebd518cca130d5f98 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e512a22ffa4800926686468233a6c3, State: Initialized, Role: FOLLOWER
I20260621 05:20:35.580163 3889 catalog_manager.cc:4841] TS 35c712dea5ca49b4a3da080f62df9adb (127.2.32.2:37907): tablet 66374d04e81e499b8c7f621e1bcd0f67 (table test-table [id=ecb4d293a37d4067b0d8170e9912b785]) successfully deleted
I20260621 05:20:35.580188 4503 consensus_queue.cc:260] T dfc06be6739e4d7ebd518cca130d5f98 P 66e512a22ffa4800926686468233a6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:35.580307 4503 ts_tablet_manager.cc:1428] T dfc06be6739e4d7ebd518cca130d5f98 P 66e512a22ffa4800926686468233a6c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:35.580415 4501 tablet_bootstrap.cc:492] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb: Bootstrap starting.
I20260621 05:20:35.580714 4501 tablet_bootstrap.cc:654] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:35.581248 4501 tablet_bootstrap.cc:492] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb: No bootstrap required, opened a new log
I20260621 05:20:35.581302 4501 ts_tablet_manager.cc:1397] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:35.581406 4501 raft_consensus.cc:348] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:35.581472 4501 raft_consensus.cc:374] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:35.581492 4501 raft_consensus.cc:729] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35c712dea5ca49b4a3da080f62df9adb, State: Initialized, Role: FOLLOWER
I20260621 05:20:35.581540 4501 consensus_queue.cc:260] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:35.581633 4501 ts_tablet_manager.cc:1428] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:35.582159 4502 tablet_bootstrap.cc:492] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30: No bootstrap required, opened a new log
I20260621 05:20:35.582212 4502 ts_tablet_manager.cc:1397] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260621 05:20:35.582640 4502 raft_consensus.cc:348] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:35.582801 4502 raft_consensus.cc:374] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:35.582826 4502 raft_consensus.cc:729] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1ed1d9dd334995b627a6cddfcf9c30, State: Initialized, Role: FOLLOWER
I20260621 05:20:35.582878 4502 consensus_queue.cc:260] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:35.582993 4502 ts_tablet_manager.cc:1428] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260621 05:20:35.598275 4249 tablet.cc:2369] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:20:35.630476 4484 raft_consensus.cc:482] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:20:35.630645 4484 raft_consensus.cc:504] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:35.630853 4484 leader_election.cc:288] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35c712dea5ca49b4a3da080f62df9adb (127.2.32.2:37907), 66e512a22ffa4800926686468233a6c3 (127.2.32.3:45001)
I20260621 05:20:35.631057 4202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc06be6739e4d7ebd518cca130d5f98" candidate_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35c712dea5ca49b4a3da080f62df9adb" is_pre_election: true
I20260621 05:20:35.631162 4342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc06be6739e4d7ebd518cca130d5f98" candidate_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e512a22ffa4800926686468233a6c3" is_pre_election: true
I20260621 05:20:35.631239 4202 raft_consensus.cc:2454] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be1ed1d9dd334995b627a6cddfcf9c30 in term 0.
I20260621 05:20:35.631249 4342 raft_consensus.cc:2454] T dfc06be6739e4d7ebd518cca130d5f98 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be1ed1d9dd334995b627a6cddfcf9c30 in term 0.
I20260621 05:20:35.631429 3995 leader_election.cc:302] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e512a22ffa4800926686468233a6c3, be1ed1d9dd334995b627a6cddfcf9c30; no voters:
I20260621 05:20:35.631557 4484 raft_consensus.cc:2781] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:20:35.631707 4484 raft_consensus.cc:482] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:20:35.631785 4484 raft_consensus.cc:3037] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:35.632699 4484 raft_consensus.cc:504] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:35.632858 4484 leader_election.cc:288] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30 [CANDIDATE]: Term 1 election: Requested vote from peers 35c712dea5ca49b4a3da080f62df9adb (127.2.32.2:37907), 66e512a22ffa4800926686468233a6c3 (127.2.32.3:45001)
I20260621 05:20:35.632992 4202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc06be6739e4d7ebd518cca130d5f98" candidate_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35c712dea5ca49b4a3da080f62df9adb"
I20260621 05:20:35.633086 4202 raft_consensus.cc:3037] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:35.633076 4342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc06be6739e4d7ebd518cca130d5f98" candidate_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e512a22ffa4800926686468233a6c3"
I20260621 05:20:35.633136 4342 raft_consensus.cc:3037] T dfc06be6739e4d7ebd518cca130d5f98 P 66e512a22ffa4800926686468233a6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:35.633867 4202 raft_consensus.cc:2454] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be1ed1d9dd334995b627a6cddfcf9c30 in term 1.
I20260621 05:20:35.633864 4342 raft_consensus.cc:2454] T dfc06be6739e4d7ebd518cca130d5f98 P 66e512a22ffa4800926686468233a6c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be1ed1d9dd334995b627a6cddfcf9c30 in term 1.
I20260621 05:20:35.634099 3993 leader_election.cc:302] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35c712dea5ca49b4a3da080f62df9adb, be1ed1d9dd334995b627a6cddfcf9c30; no voters:
I20260621 05:20:35.634218 4484 raft_consensus.cc:2781] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:35.634312 4484 raft_consensus.cc:686] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30 [term 1 LEADER]: Becoming Leader. State: Replica: be1ed1d9dd334995b627a6cddfcf9c30, State: Running, Role: LEADER
I20260621 05:20:35.634366 4484 consensus_queue.cc:237] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } }
I20260621 05:20:35.635018 3903 catalog_manager.cc:5495] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30 reported cstate change: term changed from 0 to 1, leader changed from <none> to be1ed1d9dd334995b627a6cddfcf9c30 (127.2.32.1). New cstate: current_term: 1 leader_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1ed1d9dd334995b627a6cddfcf9c30" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 45881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 } health_report { overall_health: UNKNOWN } } }
I20260621 05:20:35.652057 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:35.648628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47066 (local address 127.2.32.1:45881)
0621 05:20:35.648678 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:35.648701 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:35.648712 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:20:35.648748 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:35.648749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:35.648761 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:35.648831 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:35.649116 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.649793 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.651034 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:35.651399 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:35.651474 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:35.651718 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:35.651755 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:35.651821 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:35.651910 (+ 89us) server_negotiation.cc:299] Negotiation successful
0621 05:20:35.651936 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 05:20:35.655040 4342 raft_consensus.cc:1261] T dfc06be6739e4d7ebd518cca130d5f98 P 66e512a22ffa4800926686468233a6c3 [term 1 FOLLOWER]: Refusing update from remote peer be1ed1d9dd334995b627a6cddfcf9c30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:35.655069 4202 raft_consensus.cc:1261] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb [term 1 FOLLOWER]: Refusing update from remote peer be1ed1d9dd334995b627a6cddfcf9c30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:35.655323 4484 consensus_queue.cc:1035] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e512a22ffa4800926686468233a6c3" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:35.655452 4484 consensus_queue.cc:1035] T dfc06be6739e4d7ebd518cca130d5f98 P be1ed1d9dd334995b627a6cddfcf9c30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35c712dea5ca49b4a3da080f62df9adb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 05:20:35.660050 4472 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260621 05:20:37.186843 3904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:20:37.186954 3904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:20:37.188017 3904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c968bb12bff45e390b40346d62cf71f: Sending DeleteTablet for 3 replicas of tablet dfc06be6739e4d7ebd518cca130d5f98
I20260621 05:20:37.188315 4042 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc06be6739e4d7ebd518cca130d5f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:37 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:47050
I20260621 05:20:37.188387 4322 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc06be6739e4d7ebd518cca130d5f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:37 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:39226
I20260621 05:20:37.188537 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 3977
I20260621 05:20:37.189327 4182 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc06be6739e4d7ebd518cca130d5f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:37 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:54786
I20260621 05:20:37.189563 4555 tablet_replica.cc:330] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb: stopping tablet replica
I20260621 05:20:37.189626 4555 raft_consensus.cc:2229] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:20:37.189671 4555 raft_consensus.cc:2258] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:37.190423 4555 ts_tablet_manager.cc:1905] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:37.192392 4555 ts_tablet_manager.cc:1918] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 05:20:37.192461 4555 log.cc:1196] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/dfc06be6739e4d7ebd518cca130d5f98
I20260621 05:20:37.192670 4555 ts_tablet_manager.cc:1939] T dfc06be6739e4d7ebd518cca130d5f98 P 35c712dea5ca49b4a3da080f62df9adb: Deleting consensus metadata
I20260621 05:20:37.193053 3889 catalog_manager.cc:4841] TS 35c712dea5ca49b4a3da080f62df9adb (127.2.32.2:37907): tablet dfc06be6739e4d7ebd518cca130d5f98 (table test-table [id=f3d5f8f043d34df583ff793826fea12f]) successfully deleted
I20260621 05:20:37.195243 4554 tablet_replica.cc:330] T dfc06be6739e4d7ebd518cca130d5f98 P 66e512a22ffa4800926686468233a6c3: stopping tablet replica
I20260621 05:20:37.195326 4554 raft_consensus.cc:2229] T dfc06be6739e4d7ebd518cca130d5f98 P 66e512a22ffa4800926686468233a6c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:20:37.195425 4554 raft_consensus.cc:2258] T dfc06be6739e4d7ebd518cca130d5f98 P 66e512a22ffa4800926686468233a6c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:37.195642 4554 ts_tablet_manager.cc:1905] T dfc06be6739e4d7ebd518cca130d5f98 P 66e512a22ffa4800926686468233a6c3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260621 05:20:37.196110 3892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.32.1:45881 (error 108)
I20260621 05:20:37.196707 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:37.196486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:45881 (local address 127.0.0.1:47080)
0621 05:20:37.196624 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:37.196677 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.1:45881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260621 05:20:37.196703 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 4117
I20260621 05:20:37.197984 4554 ts_tablet_manager.cc:1918] T dfc06be6739e4d7ebd518cca130d5f98 P 66e512a22ffa4800926686468233a6c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 05:20:37.198051 4554 log.cc:1196] T dfc06be6739e4d7ebd518cca130d5f98 P 66e512a22ffa4800926686468233a6c3: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/dfc06be6739e4d7ebd518cca130d5f98
I20260621 05:20:37.198267 4554 ts_tablet_manager.cc:1939] T dfc06be6739e4d7ebd518cca130d5f98 P 66e512a22ffa4800926686468233a6c3: Deleting consensus metadata
I20260621 05:20:37.198585 3891 catalog_manager.cc:4841] TS 66e512a22ffa4800926686468233a6c3 (127.2.32.3:45001): tablet dfc06be6739e4d7ebd518cca130d5f98 (table test-table [id=f3d5f8f043d34df583ff793826fea12f]) successfully deleted
W20260621 05:20:37.203060 3892 catalog_manager.cc:4584] TS be1ed1d9dd334995b627a6cddfcf9c30 (127.2.32.1:45881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dfc06be6739e4d7ebd518cca130d5f98: Network error: Client connection negotiation failed: client connection to 127.2.32.1:45881: connect: Connection refused (error 111)
I20260621 05:20:37.203485 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 4257
I20260621 05:20:37.208968 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 3873
2026-06-21T05:20:37Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5714 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:37 dist-test-slave-b357 krb5kdc[4560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:37 dist-test-slave-b357 krb5kdc[4560](info): set up 2 sockets
Jun 21 05:20:37 dist-test-slave-b357 krb5kdc[4560](info): commencing operation
krb5kdc: starting...
W20260621 05:20:38.270956 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:20:38 dist-test-slave-b357 krb5kdc[4560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:20:38.399322 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:46881
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:46881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:38.489344 4573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:38.489560 4573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:38.489585 4573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:38.490779 4573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:38.490826 4573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:38.490842 4573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:38.490856 4573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:38.490870 4573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:38.492396 4573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:46881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:46881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:38.492590 4573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:38.492787 4573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:38.497691 4578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:38.497767 4579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:38.497721 4581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:38.497932 4573 server_base.cc:1034] running on GCE node
I20260621 05:20:38.498183 4573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:38.498463 4573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:38.499655 4573 hybrid_clock.cc:648] HybridClock initialized: now 1782019238499632 us; error 44 us; skew 500 ppm
Jun 21 05:20:38 dist-test-slave-b357 krb5kdc[4560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019238, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:38.508121 4573 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:20:38.508682 4573 webserver.cc:460] Webserver started at http://127.2.32.62:34503/ using document root <none> and password file <none>
I20260621 05:20:38.508955 4573 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:38.509032 4573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:38.509198 4573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:38.510802 4573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "21e57c228d624ea290cd51666c62fb38"
format_stamp: "Formatted at 2026-06-21 05:20:38 on dist-test-slave-b357"
server_key: "6c373da57ed8e92492b7dc2cab960db4"
server_key_iv: "25bda6701278f3bc65ad2d5ef2239898"
server_key_version: "encryptionkey@0"
I20260621 05:20:38.511379 4573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "21e57c228d624ea290cd51666c62fb38"
format_stamp: "Formatted at 2026-06-21 05:20:38 on dist-test-slave-b357"
server_key: "6c373da57ed8e92492b7dc2cab960db4"
server_key_iv: "25bda6701278f3bc65ad2d5ef2239898"
server_key_version: "encryptionkey@0"
I20260621 05:20:38.513162 4573 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260621 05:20:38.515308 4588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:38.515556 4573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260621 05:20:38.515676 4573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "21e57c228d624ea290cd51666c62fb38"
format_stamp: "Formatted at 2026-06-21 05:20:38 on dist-test-slave-b357"
server_key: "6c373da57ed8e92492b7dc2cab960db4"
server_key_iv: "25bda6701278f3bc65ad2d5ef2239898"
server_key_version: "encryptionkey@0"
I20260621 05:20:38.515795 4573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:38.542872 4573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:38.545810 4573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:38.546073 4573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:38.551417 4573 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:46881
I20260621 05:20:38.551458 4640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:46881 every 8 connection(s)
I20260621 05:20:38.551921 4573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:38.552872 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:38.554936 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 4573
I20260621 05:20:38.555059 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:20:38.555287 2176 external_mini_cluster.cc:1428] Setting key 461d178f54f2c30eb89df60681bc279e
I20260621 05:20:38.556105 4641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38: Bootstrap starting.
I20260621 05:20:38.556805 4641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:38.557207 4641 log.cc:824] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:38.558247 4641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38: No bootstrap required, opened a new log
I20260621 05:20:38.560580 4641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e57c228d624ea290cd51666c62fb38" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46881 } }
I20260621 05:20:38.560773 4641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:38.560832 4641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21e57c228d624ea290cd51666c62fb38, State: Initialized, Role: FOLLOWER
I20260621 05:20:38.560974 4641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e57c228d624ea290cd51666c62fb38" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46881 } }
I20260621 05:20:38.561053 4641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:38.561117 4641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:38.561169 4641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:38.561880 4641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e57c228d624ea290cd51666c62fb38" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46881 } }
I20260621 05:20:38.561985 4641 leader_election.cc:302] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21e57c228d624ea290cd51666c62fb38; no voters:
I20260621 05:20:38.562182 4641 leader_election.cc:288] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:38.562348 4646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:38.562561 4641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:38.562610 4646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38 [term 1 LEADER]: Becoming Leader. State: Replica: 21e57c228d624ea290cd51666c62fb38, State: Running, Role: LEADER
I20260621 05:20:38.562784 4646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e57c228d624ea290cd51666c62fb38" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46881 } }
I20260621 05:20:38.563282 4647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21e57c228d624ea290cd51666c62fb38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e57c228d624ea290cd51666c62fb38" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46881 } } }
I20260621 05:20:38.563431 4647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:38.563282 4648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21e57c228d624ea290cd51666c62fb38. Latest consensus state: current_term: 1 leader_uuid: "21e57c228d624ea290cd51666c62fb38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e57c228d624ea290cd51666c62fb38" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46881 } } }
I20260621 05:20:38.563685 4648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:38.563827 4655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:38.564442 4655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 21 05:20:38 dist-test-slave-b357 krb5kdc[4560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:38.567687 4655 catalog_manager.cc:1347] Generated new cluster ID: fc805367c8c944139d25f1b07de07231
I20260621 05:20:38.567750 4655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:38.569814 4644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:38.555862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.2.32.62:46881)
0621 05:20:38.556054 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:38.556063 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:38.556088 (+ 25us) server_negotiation.cc:407] Connection header received
0621 05:20:38.556542 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:38.556549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:38.556595 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:38.556714 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:38.557662 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:38.559515 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:38.560859 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:38.561483 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:38.561581 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:38.565601 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:38.565605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:38.565606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:38.565618 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:38.567894 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:38.568375 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:38.568377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:38.568378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:38.568484 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:38.568726 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:38.568727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:38.568728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:38.569035 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:38.569221 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:38.569328 (+ 107us) server_negotiation.cc:299] Negotiation successful
0621 05:20:38.569598 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260621 05:20:38.574062 4655 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:38.574663 4655 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:38.577658 4655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21e57c228d624ea290cd51666c62fb38: Generated new TSK 0
I20260621 05:20:38.577828 4655 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
I20260621 05:20:38.610502 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46881
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:38.699682 4667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:38.699936 4667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:38.699970 4667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:38.701262 4667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:38.701313 4667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:38.701342 4667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:20:38.702872 4667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:38.703192 4667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:38.703473 4667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:38.709415 4672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:38.709415 4673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:38.709419 4675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:38.709882 4667 server_base.cc:1034] running on GCE node
I20260621 05:20:38.710165 4667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:38.710448 4667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:38.711655 4667 hybrid_clock.cc:648] HybridClock initialized: now 1782019238711638 us; error 41 us; skew 500 ppm
Jun 21 05:20:38 dist-test-slave-b357 krb5kdc[4560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019238, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:38.714715 4667 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:20:38.715350 4667 webserver.cc:460] Webserver started at http://127.2.32.1:33119/ using document root <none> and password file <none>
I20260621 05:20:38.715623 4667 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:38.715695 4667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:38.715870 4667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:38.716977 4667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "a18904520ac740db88069c02d8a97f88"
format_stamp: "Formatted at 2026-06-21 05:20:38 on dist-test-slave-b357"
server_key: "7be3116a6b2f554075b32c1a6d46fdcb"
server_key_iv: "cf1dad97d70cf4b3aa60af2a1d6d6064"
server_key_version: "encryptionkey@0"
I20260621 05:20:38.717372 4667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "a18904520ac740db88069c02d8a97f88"
format_stamp: "Formatted at 2026-06-21 05:20:38 on dist-test-slave-b357"
server_key: "7be3116a6b2f554075b32c1a6d46fdcb"
server_key_iv: "cf1dad97d70cf4b3aa60af2a1d6d6064"
server_key_version: "encryptionkey@0"
I20260621 05:20:38.718876 4667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:20:38.719890 4682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:38.720082 4667 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:38.720184 4667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "a18904520ac740db88069c02d8a97f88"
format_stamp: "Formatted at 2026-06-21 05:20:38 on dist-test-slave-b357"
server_key: "7be3116a6b2f554075b32c1a6d46fdcb"
server_key_iv: "cf1dad97d70cf4b3aa60af2a1d6d6064"
server_key_version: "encryptionkey@0"
I20260621 05:20:38.720268 4667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:38.747480 4667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:38.750348 4667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:38.750584 4667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:38.750910 4667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:38.751308 4667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:38.751371 4667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:38.751411 4667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:38.751444 4667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:38.758220 4667 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:42647
I20260621 05:20:38.758322 4795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:42647 every 8 connection(s)
I20260621 05:20:38.758647 4667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:20:38 dist-test-slave-b357 krb5kdc[4560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019238, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:38.766206 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 4667
I20260621 05:20:38.766364 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:20:38.766571 2176 external_mini_cluster.cc:1428] Setting key 51c93b4041057f6a5f990630476cd7e1
I20260621 05:20:38.769043 4644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:38.759500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:34909 (local address 127.2.32.62:46881)
0621 05:20:38.759574 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:38.759576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:38.759729 (+ 153us) server_negotiation.cc:407] Connection header received
0621 05:20:38.760266 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:38.760267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:38.760280 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:38.760328 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:38.761282 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:38.761808 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:38.762855 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:38.763060 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:38.763163 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:38.765400 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:38.765401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:38.765402 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:38.765408 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:38.767084 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:38.767565 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:38.767567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:38.767567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:38.767625 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:38.768220 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:38.768221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:38.768222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:38.768385 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:38.768505 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:38.768817 (+ 312us) server_negotiation.cc:299] Negotiation successful
0621 05:20:38.768941 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 05:20:38.769268 4798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:38.759374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:46881 (local address 127.2.32.1:34909)
0621 05:20:38.759593 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:38.759637 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:38.760146 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:38.760392 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:38.760396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:38.760980 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:38.761178 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:38.761183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:38.761909 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:38.761933 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:38.762759 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:38.762761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:38.763166 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:38.763167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:38.763295 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:38.763829 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:38.763837 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:38.765287 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:38.767195 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:38.767206 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:38.767208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:38.767491 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:38.767839 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:38.767841 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:38.767842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:38.768108 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:38.768461 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:38.768469 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:38.768658 (+ 189us) client_negotiation.cc:769] Sending connection context
0621 05:20:38.768782 (+ 124us) client_negotiation.cc:240] Negotiation successful
0621 05:20:38.768996 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":93,"threads_started":1}
I20260621 05:20:38.770208 4796 heartbeater.cc:347] Connected to a master server at 127.2.32.62:46881
I20260621 05:20:38.770346 4796 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:38.770568 4796 heartbeater.cc:511] Master 127.2.32.62:46881 requested a full tablet report, sending...
I20260621 05:20:38.771178 4605 ts_manager.cc:194] Registered new tserver with Master: a18904520ac740db88069c02d8a97f88 (127.2.32.1:42647)
I20260621 05:20:38.771929 4605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:34909
WARNING: no policy specified for kudu/127.2.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
I20260621 05:20:38.794154 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46881
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:38.879789 4801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:38.880029 4801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:38.880064 4801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:38.881194 4801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:38.881260 4801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:38.881307 4801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:20:38.882750 4801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:38.883011 4801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:38.883260 4801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:38.889236 4806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:38.889240 4807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:38.889252 4809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:38.889626 4801 server_base.cc:1034] running on GCE node
I20260621 05:20:38.889926 4801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:38.890185 4801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:38.891376 4801 hybrid_clock.cc:648] HybridClock initialized: now 1782019238891335 us; error 70 us; skew 500 ppm
Jun 21 05:20:38 dist-test-slave-b357 krb5kdc[4560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019238, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:38.894284 4801 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:20:38.894776 4801 webserver.cc:460] Webserver started at http://127.2.32.2:40043/ using document root <none> and password file <none>
I20260621 05:20:38.894963 4801 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:38.895004 4801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:38.895092 4801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:38.896163 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "e28567a9a16547d59b87f509b5a6c7a5"
format_stamp: "Formatted at 2026-06-21 05:20:38 on dist-test-slave-b357"
server_key: "9aa9af7aa7431555e4cb23166aade519"
server_key_iv: "cbdc3e0308ef400e21f3f7132c0d46f4"
server_key_version: "encryptionkey@0"
I20260621 05:20:38.896458 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "e28567a9a16547d59b87f509b5a6c7a5"
format_stamp: "Formatted at 2026-06-21 05:20:38 on dist-test-slave-b357"
server_key: "9aa9af7aa7431555e4cb23166aade519"
server_key_iv: "cbdc3e0308ef400e21f3f7132c0d46f4"
server_key_version: "encryptionkey@0"
I20260621 05:20:38.897716 4801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:20:38.898419 4816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:38.898641 4801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:38.898731 4801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "e28567a9a16547d59b87f509b5a6c7a5"
format_stamp: "Formatted at 2026-06-21 05:20:38 on dist-test-slave-b357"
server_key: "9aa9af7aa7431555e4cb23166aade519"
server_key_iv: "cbdc3e0308ef400e21f3f7132c0d46f4"
server_key_version: "encryptionkey@0"
I20260621 05:20:38.898784 4801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:38.911553 4801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:38.913699 4801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:38.913807 4801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:38.913964 4801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:38.914266 4801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:38.914301 4801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:38.914322 4801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:38.914335 4801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:38.920744 4801 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:42427
I20260621 05:20:38.920871 4929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:42427 every 8 connection(s)
I20260621 05:20:38.921149 4801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:20:38 dist-test-slave-b357 krb5kdc[4560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019238, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:38.929785 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 4801
I20260621 05:20:38.929909 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:20:38.930084 2176 external_mini_cluster.cc:1428] Setting key b08385508d693f7fcee1093c4087cf33
I20260621 05:20:38.932446 4644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:38.921837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:59535 (local address 127.2.32.62:46881)
0621 05:20:38.921879 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:38.921881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:38.922113 (+ 232us) server_negotiation.cc:407] Connection header received
0621 05:20:38.922611 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:38.922612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:38.922622 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:38.922686 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:38.923799 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:38.924413 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:38.925704 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:38.925901 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:38.925974 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:38.928639 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:38.928641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:38.928641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:38.928650 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:38.930572 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:38.931058 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:38.931059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:38.931060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:38.931155 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:38.931411 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:38.931412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:38.931412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:38.931570 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:38.931643 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:38.932246 (+ 603us) server_negotiation.cc:299] Negotiation successful
0621 05:20:38.932361 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:20:38.933005 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:38.921811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:46881 (local address 127.2.32.2:59535)
0621 05:20:38.921968 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:38.922007 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:38.922479 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:38.922763 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:38.922766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:38.923414 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:38.923605 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:38.923610 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:38.924628 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:38.924629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:38.925585 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:38.925588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:38.926027 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:38.926029 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:38.926254 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:38.926886 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:38.926894 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:38.928510 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:38.930652 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:38.930658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:38.930661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:38.930981 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:38.931245 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:38.931245 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:38.931246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:38.931343 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:38.931683 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:38.931694 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:38.931993 (+ 299us) client_negotiation.cc:769] Sending connection context
0621 05:20:38.932596 (+ 603us) client_negotiation.cc:240] Negotiation successful
0621 05:20:38.932795 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":95,"threads_started":1}
I20260621 05:20:38.933702 4930 heartbeater.cc:347] Connected to a master server at 127.2.32.62:46881
I20260621 05:20:38.933822 4930 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:38.934017 4930 heartbeater.cc:511] Master 127.2.32.62:46881 requested a full tablet report, sending...
I20260621 05:20:38.934399 4605 ts_manager.cc:194] Registered new tserver with Master: e28567a9a16547d59b87f509b5a6c7a5 (127.2.32.2:42427)
I20260621 05:20:38.934787 4605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:59535
WARNING: no policy specified for kudu/127.2.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
I20260621 05:20:38.957904 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46881
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:39.045249 4935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:39.045482 4935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:39.045518 4935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:39.046633 4935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:39.046703 4935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:39.046746 4935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:20:39.048233 4935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:39.048504 4935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:39.048730 4935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:39.053923 4940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:39.053931 4941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:39.053963 4935 server_base.cc:1034] running on GCE node
W20260621 05:20:39.053931 4943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:39.054312 4935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:39.054555 4935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:39.055728 4935 hybrid_clock.cc:648] HybridClock initialized: now 1782019239055718 us; error 33 us; skew 500 ppm
Jun 21 05:20:39 dist-test-slave-b357 krb5kdc[4560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019239, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:39.058472 4935 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:20:39.058878 4935 webserver.cc:460] Webserver started at http://127.2.32.3:34989/ using document root <none> and password file <none>
I20260621 05:20:39.059105 4935 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:39.059221 4935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:39.059326 4935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:39.060623 4935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "6b8e59fa0d3c4bcf83872795ea2008ee"
format_stamp: "Formatted at 2026-06-21 05:20:39 on dist-test-slave-b357"
server_key: "3ac265a19030a5c98908329af863fedb"
server_key_iv: "7d36d0718180f60c5e44ef73af4634ab"
server_key_version: "encryptionkey@0"
I20260621 05:20:39.061097 4935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "6b8e59fa0d3c4bcf83872795ea2008ee"
format_stamp: "Formatted at 2026-06-21 05:20:39 on dist-test-slave-b357"
server_key: "3ac265a19030a5c98908329af863fedb"
server_key_iv: "7d36d0718180f60c5e44ef73af4634ab"
server_key_version: "encryptionkey@0"
I20260621 05:20:39.062713 4935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:20:39.064078 4950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:39.064419 4935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:39.064549 4935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "6b8e59fa0d3c4bcf83872795ea2008ee"
format_stamp: "Formatted at 2026-06-21 05:20:39 on dist-test-slave-b357"
server_key: "3ac265a19030a5c98908329af863fedb"
server_key_iv: "7d36d0718180f60c5e44ef73af4634ab"
server_key_version: "encryptionkey@0"
I20260621 05:20:39.064625 4935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:39.074568 4935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:39.076957 4935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:39.077072 4935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:39.077253 4935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:39.077585 4935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:39.077760 4935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:39.077821 4935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:39.077842 4935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:39.084175 4935 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:35891
I20260621 05:20:39.084295 5063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:35891 every 8 connection(s)
I20260621 05:20:39.084614 4935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:20:39 dist-test-slave-b357 krb5kdc[4560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019239, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:39.093128 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 4935
I20260621 05:20:39.093250 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:20:39.093449 2176 external_mini_cluster.cc:1428] Setting key 10e84f8bba1a8fe3a32218b0d249d4f1
I20260621 05:20:39.095803 4644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:39.085332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:54405 (local address 127.2.32.62:46881)
0621 05:20:39.085418 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:39.085421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:39.085602 (+ 181us) server_negotiation.cc:407] Connection header received
0621 05:20:39.086113 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:39.086114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:39.086124 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:39.086183 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:39.087228 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:39.087807 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:39.088992 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:39.089204 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:39.089286 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:39.092318 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:39.092320 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:39.092321 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:39.092328 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:39.094311 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:39.094796 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:39.094796 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:39.094797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:39.094852 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:39.095085 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:39.095086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:39.095087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:39.095276 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:39.095333 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:39.095659 (+ 326us) server_negotiation.cc:299] Negotiation successful
0621 05:20:39.095736 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 05:20:39.095993 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:39.085298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:46881 (local address 127.2.32.3:54405)
0621 05:20:39.085479 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:39.085515 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:39.086006 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:39.086235 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:39.086238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:39.086805 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:39.086998 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:39.087003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:39.087938 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:39.087939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:39.088818 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:39.088820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:39.089320 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:39.089321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:39.089476 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:39.090172 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:39.090185 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:39.092177 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:39.094404 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:39.094411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:39.094413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:39.094723 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:39.094915 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:39.094915 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:39.094916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:39.095021 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:39.095358 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:39.095364 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:39.095507 (+ 143us) client_negotiation.cc:769] Sending connection context
0621 05:20:39.095610 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 05:20:39.095757 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":106,"threads_started":1}
I20260621 05:20:39.096616 5064 heartbeater.cc:347] Connected to a master server at 127.2.32.62:46881
I20260621 05:20:39.096714 5064 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:39.096997 5064 heartbeater.cc:511] Master 127.2.32.62:46881 requested a full tablet report, sending...
I20260621 05:20:39.097446 4605 ts_manager.cc:194] Registered new tserver with Master: 6b8e59fa0d3c4bcf83872795ea2008ee (127.2.32.3:35891)
I20260621 05:20:39.097848 4605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:54405
I20260621 05:20:39.105739 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 05:20:39.117094 4644 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:39.116158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37408 (local address 127.2.32.62:46881)
0621 05:20:39.116240 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:39.116241 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:39.116334 (+ 93us) server_negotiation.cc:407] Connection header received
0621 05:20:39.116376 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:39.116376 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:39.116385 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:39.116446 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:39.117028 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37408: BlockingRecv error: recv got EOF from 127.0.0.1:37408 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 05:20:39.118048 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 4667
I20260621 05:20:39.122661 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 4801
I20260621 05:20:39.128289 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 4935
I20260621 05:20:39.133559 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 4573
2026-06-21T05:20:39Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1922 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:39 dist-test-slave-b357 krb5kdc[5076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:39 dist-test-slave-b357 krb5kdc[5076](info): set up 2 sockets
Jun 21 05:20:39 dist-test-slave-b357 krb5kdc[5076](info): commencing operation
krb5kdc: starting...
W20260621 05:20:40.191752 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:20:40 dist-test-slave-b357 krb5kdc[5076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:20:40.318914 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:39191
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:39191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:40.401952 5089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:40.402146 5089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:40.402168 5089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:40.403369 5089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:40.403412 5089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:40.403427 5089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:40.403440 5089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:40.403452 5089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:40.404896 5089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:39191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:39191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:40.405117 5089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:40.405359 5089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:40.410859 5094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:40.411063 5097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:40.411296 5095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:40.411484 5089 server_base.cc:1034] running on GCE node
I20260621 05:20:40.411799 5089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:40.412144 5089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:40.413352 5089 hybrid_clock.cc:648] HybridClock initialized: now 1782019240413312 us; error 68 us; skew 500 ppm
Jun 21 05:20:40 dist-test-slave-b357 krb5kdc[5076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019240, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:40.416452 5089 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:20:40.416929 5089 webserver.cc:460] Webserver started at http://127.2.32.62:37525/ using document root <none> and password file <none>
I20260621 05:20:40.417203 5089 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:40.417272 5089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:40.417423 5089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:40.418450 5089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "2d14f3b5794d4f6b9cc6196f2d59dd28"
format_stamp: "Formatted at 2026-06-21 05:20:40 on dist-test-slave-b357"
server_key: "ca8164de8bd97e06bc8baad6aa74e9ba"
server_key_iv: "f0a0b367690b231476d457f64868d6f6"
server_key_version: "encryptionkey@0"
I20260621 05:20:40.418817 5089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "2d14f3b5794d4f6b9cc6196f2d59dd28"
format_stamp: "Formatted at 2026-06-21 05:20:40 on dist-test-slave-b357"
server_key: "ca8164de8bd97e06bc8baad6aa74e9ba"
server_key_iv: "f0a0b367690b231476d457f64868d6f6"
server_key_version: "encryptionkey@0"
I20260621 05:20:40.420223 5089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:20:40.420992 5104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:40.421281 5089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:40.421465 5089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "2d14f3b5794d4f6b9cc6196f2d59dd28"
format_stamp: "Formatted at 2026-06-21 05:20:40 on dist-test-slave-b357"
server_key: "ca8164de8bd97e06bc8baad6aa74e9ba"
server_key_iv: "f0a0b367690b231476d457f64868d6f6"
server_key_version: "encryptionkey@0"
I20260621 05:20:40.421583 5089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:40.443102 5089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:40.445479 5089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:40.445652 5089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:40.449822 5089 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:39191
I20260621 05:20:40.449891 5156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:39191 every 8 connection(s)
I20260621 05:20:40.450244 5089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:40.451198 5157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:40.453447 5157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28: Bootstrap starting.
I20260621 05:20:40.454090 5157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:40.454421 5157 log.cc:824] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:40.454818 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 5089
I20260621 05:20:40.454900 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:20:40.455104 2176 external_mini_cluster.cc:1428] Setting key e0ab4ef4a1f3542c96a180fc805ec390
I20260621 05:20:40.455300 5157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28: No bootstrap required, opened a new log
I20260621 05:20:40.458066 5157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d14f3b5794d4f6b9cc6196f2d59dd28" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 39191 } }
I20260621 05:20:40.458246 5157 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:40.458272 5157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d14f3b5794d4f6b9cc6196f2d59dd28, State: Initialized, Role: FOLLOWER
I20260621 05:20:40.458475 5157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d14f3b5794d4f6b9cc6196f2d59dd28" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 39191 } }
I20260621 05:20:40.458568 5157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:40.458608 5157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:40.458659 5157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:40.459581 5157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d14f3b5794d4f6b9cc6196f2d59dd28" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 39191 } }
I20260621 05:20:40.459750 5157 leader_election.cc:302] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d14f3b5794d4f6b9cc6196f2d59dd28; no voters:
I20260621 05:20:40.459959 5157 leader_election.cc:288] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:40.459986 5162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:40.460094 5162 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28 [term 1 LEADER]: Becoming Leader. State: Replica: 2d14f3b5794d4f6b9cc6196f2d59dd28, State: Running, Role: LEADER
I20260621 05:20:40.460216 5157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:40.460194 5162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d14f3b5794d4f6b9cc6196f2d59dd28" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 39191 } }
I20260621 05:20:40.460646 5164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d14f3b5794d4f6b9cc6196f2d59dd28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d14f3b5794d4f6b9cc6196f2d59dd28" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 39191 } } }
I20260621 05:20:40.460652 5163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d14f3b5794d4f6b9cc6196f2d59dd28. Latest consensus state: current_term: 1 leader_uuid: "2d14f3b5794d4f6b9cc6196f2d59dd28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d14f3b5794d4f6b9cc6196f2d59dd28" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 39191 } } }
I20260621 05:20:40.460848 5164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:40.460866 5163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:40.461486 5171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:40.462061 5171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 21 05:20:40 dist-test-slave-b357 krb5kdc[5076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:40.464243 5171 catalog_manager.cc:1347] Generated new cluster ID: 6e0969f992954055a4d5c13b1a054aa0
I20260621 05:20:40.464313 5171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:40.468190 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:40.455674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57132 (local address 127.2.32.62:39191)
0621 05:20:40.455869 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:40.455879 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:40.455913 (+ 34us) server_negotiation.cc:407] Connection header received
0621 05:20:40.456711 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:40.456725 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:40.456812 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:40.457102 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:40.457833 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:40.459096 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:40.460193 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:40.460908 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:40.461106 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:40.464654 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:40.464657 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:40.464657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:40.464669 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:40.466540 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:40.466950 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:40.466952 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:40.466952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:40.467057 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:40.467288 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:40.467289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:40.467289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:40.467569 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:40.467711 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:40.467892 (+ 181us) server_negotiation.cc:299] Negotiation successful
0621 05:20:40.468013 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260621 05:20:40.469244 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:20:40.488335 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:40.483913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.2.32.62:39191)
0621 05:20:40.483961 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:40.483963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:40.484095 (+ 132us) server_negotiation.cc:407] Connection header received
0621 05:20:40.484176 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:40.484177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:40.484188 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:40.484244 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:40.484679 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:40.485689 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:40.486724 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:40.487033 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:40.487100 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:40.487854 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:40.487855 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:40.487855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:40.488054 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:40.488070 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:40.488076 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:40.488143 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:40.488234 (+ 91us) server_negotiation.cc:299] Negotiation successful
0621 05:20:40.488257 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:20:40.489993 5171 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:40.490705 5171 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:40.497047 5171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d14f3b5794d4f6b9cc6196f2d59dd28: Generated new TSK 0
I20260621 05:20:40.497221 5171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:20:40.500068 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 5089
2026-06-21T05:20:40Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1365 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:40 dist-test-slave-b357 krb5kdc[5190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:40 dist-test-slave-b357 krb5kdc[5190](info): set up 2 sockets
Jun 21 05:20:40 dist-test-slave-b357 krb5kdc[5190](info): commencing operation
krb5kdc: starting...
W20260621 05:20:41.551045 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:20:41 dist-test-slave-b357 krb5kdc[5190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:20:41.678444 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:41477
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:33289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:41477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:41.769330 5203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:41.769608 5203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:41.769635 5203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:41.771719 5203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:41.771808 5203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:41.771828 5203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:41.771852 5203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:41.771867 5203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:41.774451 5203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:33289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:41477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:41477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:41.774729 5203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:41.774993 5203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:41.780138 5211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:41.780121 5208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:41.780426 5203 server_base.cc:1034] running on GCE node
W20260621 05:20:41.780128 5209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:41.780752 5203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:41.781059 5203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:41.782284 5203 hybrid_clock.cc:648] HybridClock initialized: now 1782019241782253 us; error 47 us; skew 500 ppm
Jun 21 05:20:41 dist-test-slave-b357 krb5kdc[5190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019241, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:41.784946 5203 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:20:41.785440 5203 webserver.cc:460] Webserver started at http://127.2.32.62:35567/ using document root <none> and password file <none>
I20260621 05:20:41.785633 5203 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:41.785681 5203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:41.785772 5203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:41.786806 5203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "ca474a69f1b5444cba493f2f22249b3f"
format_stamp: "Formatted at 2026-06-21 05:20:41 on dist-test-slave-b357"
server_key: "fd2f1d71255f3d7e3646b7a9ab5ef487"
server_key_iv: "7781294d60bbb9ff0b85d91440b2b599"
server_key_version: "encryptionkey@0"
I20260621 05:20:41.787194 5203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "ca474a69f1b5444cba493f2f22249b3f"
format_stamp: "Formatted at 2026-06-21 05:20:41 on dist-test-slave-b357"
server_key: "fd2f1d71255f3d7e3646b7a9ab5ef487"
server_key_iv: "7781294d60bbb9ff0b85d91440b2b599"
server_key_version: "encryptionkey@0"
I20260621 05:20:41.788790 5203 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:41.789641 5218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:41.789880 5203 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:41.789968 5203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "ca474a69f1b5444cba493f2f22249b3f"
format_stamp: "Formatted at 2026-06-21 05:20:41 on dist-test-slave-b357"
server_key: "fd2f1d71255f3d7e3646b7a9ab5ef487"
server_key_iv: "7781294d60bbb9ff0b85d91440b2b599"
server_key_version: "encryptionkey@0"
I20260621 05:20:41.790020 5203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:41.805006 5203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:41.807564 5203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:41.807684 5203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:41.811456 5203 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:41477
I20260621 05:20:41.811538 5270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:41477 every 8 connection(s)
I20260621 05:20:41.811857 5203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:41.812988 5271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:41.814754 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 5203
I20260621 05:20:41.814836 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:20:41.815115 2176 external_mini_cluster.cc:1428] Setting key d705375b0f7517541c6c9d838174dead
I20260621 05:20:41.815946 5271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f: Bootstrap starting.
I20260621 05:20:41.816879 5271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:41.817236 5271 log.cc:824] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:41.817960 5271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f: No bootstrap required, opened a new log
I20260621 05:20:41.820241 5271 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca474a69f1b5444cba493f2f22249b3f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 41477 } }
I20260621 05:20:41.820401 5271 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:41.820433 5271 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca474a69f1b5444cba493f2f22249b3f, State: Initialized, Role: FOLLOWER
I20260621 05:20:41.820545 5271 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca474a69f1b5444cba493f2f22249b3f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 41477 } }
I20260621 05:20:41.820629 5271 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:41.820659 5271 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:41.820703 5271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:41.821444 5271 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca474a69f1b5444cba493f2f22249b3f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 41477 } }
I20260621 05:20:41.821529 5271 leader_election.cc:302] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca474a69f1b5444cba493f2f22249b3f; no voters:
I20260621 05:20:41.821708 5271 leader_election.cc:288] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:41.821902 5276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:41.821964 5271 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:41.822189 5276 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f [term 1 LEADER]: Becoming Leader. State: Replica: ca474a69f1b5444cba493f2f22249b3f, State: Running, Role: LEADER
I20260621 05:20:41.822386 5276 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca474a69f1b5444cba493f2f22249b3f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 41477 } }
I20260621 05:20:41.823108 5278 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca474a69f1b5444cba493f2f22249b3f. Latest consensus state: current_term: 1 leader_uuid: "ca474a69f1b5444cba493f2f22249b3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca474a69f1b5444cba493f2f22249b3f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 41477 } } }
I20260621 05:20:41.823287 5278 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:41.823343 5277 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca474a69f1b5444cba493f2f22249b3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca474a69f1b5444cba493f2f22249b3f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 41477 } } }
I20260621 05:20:41.823446 5277 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:41.823566 5289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 21 05:20:41 dist-test-slave-b357 krb5kdc[5190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:41.824772 5289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:41.827106 5289 catalog_manager.cc:1347] Generated new cluster ID: c5f46b9c376f44468c7d5135cd16f748
I20260621 05:20:41.827265 5289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:41.828819 5274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:41.815863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51416 (local address 127.2.32.62:41477)
0621 05:20:41.816192 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:41.816204 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:41.816241 (+ 37us) server_negotiation.cc:407] Connection header received
0621 05:20:41.817058 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:41.817069 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:41.817133 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:41.817287 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:41.817934 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:41.819403 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:41.820189 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:41.820781 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:41.820890 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:41.824537 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:41.824541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:41.824542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:41.824562 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:41.826737 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:41.827249 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:41.827254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:41.827254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:41.827411 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:41.827711 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:41.827712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:41.827712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:41.828049 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:41.828195 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:41.828417 (+ 222us) server_negotiation.cc:299] Negotiation successful
0621 05:20:41.828582 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":186,"threads_started":1}
I20260621 05:20:41.856848 5289 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:41.857424 5289 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:41.863518 5289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca474a69f1b5444cba493f2f22249b3f: Generated new TSK 0
I20260621 05:20:41.863811 5289 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
I20260621 05:20:41.894238 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:41477
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:33289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:42.007257 5297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:42.007604 5297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:42.007651 5297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:42.010121 5297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:42.010185 5297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:42.010227 5297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:20:42.012591 5297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:33289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:41477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:42.012895 5297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:42.013161 5297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:42.019471 5303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:42.019449 5302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:42.019608 5297 server_base.cc:1034] running on GCE node
W20260621 05:20:42.019767 5305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:42.020002 5297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:42.020637 5297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:42.021953 5297 hybrid_clock.cc:648] HybridClock initialized: now 1782019242021900 us; error 44 us; skew 500 ppm
Jun 21 05:20:42 dist-test-slave-b357 krb5kdc[5190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019242, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:42.025621 5297 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:20:42.026243 5297 webserver.cc:460] Webserver started at http://127.2.32.1:37951/ using document root <none> and password file <none>
I20260621 05:20:42.026542 5297 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:42.026615 5297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:42.026753 5297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:42.028329 5297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "d752dcf1f5524367a66b2fb708c866b7"
format_stamp: "Formatted at 2026-06-21 05:20:42 on dist-test-slave-b357"
server_key: "494a6132ea444bc109df2393f892f272"
server_key_iv: "6bdda1705e7ce0c132e5f825ef3974a2"
server_key_version: "encryptionkey@0"
I20260621 05:20:42.028764 5297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "d752dcf1f5524367a66b2fb708c866b7"
format_stamp: "Formatted at 2026-06-21 05:20:42 on dist-test-slave-b357"
server_key: "494a6132ea444bc109df2393f892f272"
server_key_iv: "6bdda1705e7ce0c132e5f825ef3974a2"
server_key_version: "encryptionkey@0"
I20260621 05:20:42.030544 5297 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260621 05:20:42.031657 5312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:42.031921 5297 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:20:42.032058 5297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "d752dcf1f5524367a66b2fb708c866b7"
format_stamp: "Formatted at 2026-06-21 05:20:42 on dist-test-slave-b357"
server_key: "494a6132ea444bc109df2393f892f272"
server_key_iv: "6bdda1705e7ce0c132e5f825ef3974a2"
server_key_version: "encryptionkey@0"
I20260621 05:20:42.032179 5297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:42.055815 5297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:42.059237 5297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:42.059396 5297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:42.059696 5297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:42.060108 5297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:42.060169 5297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:42.060205 5297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:42.060223 5297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:42.066479 5297 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:35313
I20260621 05:20:42.066570 5425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:35313 every 8 connection(s)
I20260621 05:20:42.066877 5297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:20:42.070508 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 5297
I20260621 05:20:42.070593 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:20:42.070806 2176 external_mini_cluster.cc:1428] Setting key 63604b18c06e61eb23f509b9d2b8d858
Jun 21 05:20:42 dist-test-slave-b357 krb5kdc[5190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019242, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:42.076632 5428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:42.067647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:41477 (local address 127.2.32.1:50431)
0621 05:20:42.067818 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:42.067859 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:42.068330 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:42.068563 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:42.068566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:42.069108 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:42.069289 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:42.069293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:42.070053 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:42.070053 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:42.070998 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:42.071001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:42.071519 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:42.071520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:42.071638 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:42.072135 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:42.072143 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:42.073459 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:42.075116 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:42.075123 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:42.075145 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:42.075466 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:42.075697 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:42.075698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:42.075698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:42.075795 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:42.076135 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:42.076142 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:42.076256 (+ 114us) client_negotiation.cc:769] Sending connection context
0621 05:20:42.076351 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 05:20:42.076468 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260621 05:20:42.076648 5274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:42.067725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:50431 (local address 127.2.32.62:41477)
0621 05:20:42.067779 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:42.067781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:42.067970 (+ 189us) server_negotiation.cc:407] Connection header received
0621 05:20:42.068431 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:42.068432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:42.068443 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:42.068506 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:42.069400 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:42.069959 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:42.071079 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:42.071424 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:42.071505 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:42.073545 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:42.073546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:42.073547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:42.073555 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:42.075025 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:42.075560 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:42.075561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:42.075562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:42.075623 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:42.075867 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:42.075868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:42.075869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:42.076039 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:42.076120 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:42.076400 (+ 280us) server_negotiation.cc:299] Negotiation successful
0621 05:20:42.076503 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260621 05:20:42.077162 5426 heartbeater.cc:347] Connected to a master server at 127.2.32.62:41477
I20260621 05:20:42.077267 5426 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:42.077479 5426 heartbeater.cc:511] Master 127.2.32.62:41477 requested a full tablet report, sending...
I20260621 05:20:42.078061 5235 ts_manager.cc:194] Registered new tserver with Master: d752dcf1f5524367a66b2fb708c866b7 (127.2.32.1:35313)
I20260621 05:20:42.078682 5235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:50431
WARNING: no policy specified for kudu/127.2.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
I20260621 05:20:42.099088 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:41477
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:33289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:42.182770 5431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:42.182963 5431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:42.182991 5431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:42.184239 5431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:42.184283 5431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:42.184309 5431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:20:42.185706 5431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:33289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:41477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:42.185953 5431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:42.186184 5431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:42.191234 5437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:42.191283 5436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:42.191450 5431 server_base.cc:1034] running on GCE node
W20260621 05:20:42.191283 5439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:42.191741 5431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:42.191995 5431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:42.193187 5431 hybrid_clock.cc:648] HybridClock initialized: now 1782019242193156 us; error 51 us; skew 500 ppm
Jun 21 05:20:42 dist-test-slave-b357 krb5kdc[5190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019242, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:42.195940 5431 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:20:42.196403 5431 webserver.cc:460] Webserver started at http://127.2.32.2:36649/ using document root <none> and password file <none>
I20260621 05:20:42.196621 5431 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:42.196688 5431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:42.196842 5431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:42.197857 5431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "2247578c608347b9a4c974cfcf77d5c0"
format_stamp: "Formatted at 2026-06-21 05:20:42 on dist-test-slave-b357"
server_key: "b119631d05c19a9f46e001961d139b03"
server_key_iv: "e4e2d122ed69bc7870104f99cad4743e"
server_key_version: "encryptionkey@0"
I20260621 05:20:42.198218 5431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "2247578c608347b9a4c974cfcf77d5c0"
format_stamp: "Formatted at 2026-06-21 05:20:42 on dist-test-slave-b357"
server_key: "b119631d05c19a9f46e001961d139b03"
server_key_iv: "e4e2d122ed69bc7870104f99cad4743e"
server_key_version: "encryptionkey@0"
I20260621 05:20:42.199597 5431 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:20:42.200493 5446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:42.200809 5431 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:42.201027 5431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "2247578c608347b9a4c974cfcf77d5c0"
format_stamp: "Formatted at 2026-06-21 05:20:42 on dist-test-slave-b357"
server_key: "b119631d05c19a9f46e001961d139b03"
server_key_iv: "e4e2d122ed69bc7870104f99cad4743e"
server_key_version: "encryptionkey@0"
I20260621 05:20:42.201135 5431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:42.210737 5431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:42.213120 5431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:42.213292 5431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:42.213897 5431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:42.214323 5431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:42.214386 5431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:42.214434 5431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:42.214459 5431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:42.221060 5431 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:34351
I20260621 05:20:42.221138 5559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:34351 every 8 connection(s)
I20260621 05:20:42.221474 5431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:20:42.224556 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 5431
I20260621 05:20:42.224673 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:20:42.224850 2176 external_mini_cluster.cc:1428] Setting key 9b3349372febb0b56cca2bbc3739b129
Jun 21 05:20:42 dist-test-slave-b357 krb5kdc[5190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019242, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:42.233212 5274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:42.222278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:44245 (local address 127.2.32.62:41477)
0621 05:20:42.222347 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:42.222349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:42.222496 (+ 147us) server_negotiation.cc:407] Connection header received
0621 05:20:42.223171 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:42.223175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:42.223195 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:42.223293 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:42.224273 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:42.225180 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:42.226482 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:42.226885 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:42.226981 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:42.229645 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:42.229647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:42.229648 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:42.229658 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:42.231707 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:42.232208 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:42.232210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:42.232210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:42.232297 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:42.232573 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:42.232574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:42.232575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:42.232691 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:42.232764 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:42.233037 (+ 273us) server_negotiation.cc:299] Negotiation successful
0621 05:20:42.233121 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 05:20:42.233393 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:42.222200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:41477 (local address 127.2.32.2:44245)
0621 05:20:42.222358 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:42.222397 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:42.223015 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:42.223345 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:42.223349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:42.223936 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:42.224120 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:42.224124 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:42.225523 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:42.225524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:42.226372 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:42.226375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:42.227057 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:42.227058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:42.227244 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:42.227840 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:42.227848 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:42.229529 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:42.231905 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:42.231911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:42.231912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:42.232134 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:42.232381 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:42.232382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:42.232383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:42.232493 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:42.232777 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:42.232784 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:42.232909 (+ 125us) client_negotiation.cc:769] Sending connection context
0621 05:20:42.233016 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 05:20:42.233169 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260621 05:20:42.233788 5560 heartbeater.cc:347] Connected to a master server at 127.2.32.62:41477
I20260621 05:20:42.233876 5560 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:42.234052 5560 heartbeater.cc:511] Master 127.2.32.62:41477 requested a full tablet report, sending...
I20260621 05:20:42.234408 5235 ts_manager.cc:194] Registered new tserver with Master: 2247578c608347b9a4c974cfcf77d5c0 (127.2.32.2:34351)
I20260621 05:20:42.234795 5235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:44245
WARNING: no policy specified for kudu/127.2.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
I20260621 05:20:42.254992 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:41477
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:33289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:42.343114 5565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:42.343426 5565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:42.343458 5565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:42.344683 5565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:42.344727 5565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:42.344756 5565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:20:42.346201 5565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:33289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:41477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:42.346472 5565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:42.346704 5565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:42.351606 5573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:42.351805 5570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:42.351903 5571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:42.352200 5565 server_base.cc:1034] running on GCE node
I20260621 05:20:42.352435 5565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:42.352701 5565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:42.353881 5565 hybrid_clock.cc:648] HybridClock initialized: now 1782019242353857 us; error 39 us; skew 500 ppm
Jun 21 05:20:42 dist-test-slave-b357 krb5kdc[5190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019242, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:42.356833 5565 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:20:42.357259 5565 webserver.cc:460] Webserver started at http://127.2.32.3:35683/ using document root <none> and password file <none>
I20260621 05:20:42.357507 5565 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:42.357607 5565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:42.357738 5565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:42.358830 5565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "4dbd641e25334c269b77c386fee9eb8f"
format_stamp: "Formatted at 2026-06-21 05:20:42 on dist-test-slave-b357"
server_key: "0b66689fc2fac1616917f64a1c5c2633"
server_key_iv: "6fbd739cabab5b4c84a04e1a71e09069"
server_key_version: "encryptionkey@0"
I20260621 05:20:42.359192 5565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "4dbd641e25334c269b77c386fee9eb8f"
format_stamp: "Formatted at 2026-06-21 05:20:42 on dist-test-slave-b357"
server_key: "0b66689fc2fac1616917f64a1c5c2633"
server_key_iv: "6fbd739cabab5b4c84a04e1a71e09069"
server_key_version: "encryptionkey@0"
I20260621 05:20:42.360704 5565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:42.361733 5580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:42.362104 5565 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:42.362207 5565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "4dbd641e25334c269b77c386fee9eb8f"
format_stamp: "Formatted at 2026-06-21 05:20:42 on dist-test-slave-b357"
server_key: "0b66689fc2fac1616917f64a1c5c2633"
server_key_iv: "6fbd739cabab5b4c84a04e1a71e09069"
server_key_version: "encryptionkey@0"
I20260621 05:20:42.362421 5565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:42.372020 5565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:42.374265 5565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:42.374447 5565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:42.374693 5565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:42.375052 5565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:42.375100 5565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:42.375214 5565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:42.375267 5565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:42.382431 5565 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:35485
I20260621 05:20:42.382475 5693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:35485 every 8 connection(s)
I20260621 05:20:42.382869 5565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:20:42 dist-test-slave-b357 krb5kdc[5190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019242, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:42.390103 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 5565
I20260621 05:20:42.390229 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:20:42.390493 2176 external_mini_cluster.cc:1428] Setting key 214c42b5e8d0eb4b433ddc6036760c19
I20260621 05:20:42.393504 5274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:42.383804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:38279 (local address 127.2.32.62:41477)
0621 05:20:42.383855 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:42.383857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:42.384121 (+ 264us) server_negotiation.cc:407] Connection header received
0621 05:20:42.384676 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:42.384678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:42.384691 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:42.384767 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:42.385715 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:42.386285 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:42.387331 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:42.387643 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:42.387730 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:42.390304 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:42.390307 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:42.390307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:42.390325 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:42.391853 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:42.392357 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:42.392358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:42.392359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:42.392433 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:42.392691 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:42.392692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:42.392692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:42.392796 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:42.392851 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:42.393352 (+ 501us) server_negotiation.cc:299] Negotiation successful
0621 05:20:42.393430 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:20:42.393821 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:42.383756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:41477 (local address 127.2.32.3:38279)
0621 05:20:42.383981 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:42.384019 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:42.384562 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:42.384836 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:42.384842 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:42.385415 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:42.385608 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:42.385613 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:42.386382 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:42.386383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:42.387226 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:42.387228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:42.387737 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:42.387739 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:42.387865 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:42.388418 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:42.388426 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:42.390109 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:42.391965 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:42.391971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:42.391973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:42.392250 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:42.392516 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:42.392517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:42.392517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:42.392630 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:42.392915 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:42.392922 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:42.393052 (+ 130us) client_negotiation.cc:769] Sending connection context
0621 05:20:42.393293 (+ 241us) client_negotiation.cc:240] Negotiation successful
0621 05:20:42.393610 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":89,"threads_started":1}
I20260621 05:20:42.394402 5694 heartbeater.cc:347] Connected to a master server at 127.2.32.62:41477
I20260621 05:20:42.394500 5694 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:42.394639 5694 heartbeater.cc:511] Master 127.2.32.62:41477 requested a full tablet report, sending...
I20260621 05:20:42.395044 5235 ts_manager.cc:194] Registered new tserver with Master: 4dbd641e25334c269b77c386fee9eb8f (127.2.32.3:35485)
I20260621 05:20:42.395453 5235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:38279
I20260621 05:20:42.402228 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 05:20:42 dist-test-slave-b357 krb5kdc[5190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019242, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 21 05:20:42 dist-test-slave-b357 krb5kdc[5190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019242, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:42.431357 5274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:42.421816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51432 (local address 127.2.32.62:41477)
0621 05:20:42.421869 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:42.421871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:42.421997 (+ 126us) server_negotiation.cc:407] Connection header received
0621 05:20:42.422086 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:42.422087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:42.422097 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:42.422153 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:42.423028 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:42.423824 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:42.425016 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:42.425219 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:42.425297 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:42.428271 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:42.428273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:42.428273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:42.428282 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:42.430000 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:42.430454 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:42.430455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:42.430455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:42.430528 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:42.430723 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:42.430723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:42.430723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:42.430875 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:42.430961 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:42.431103 (+ 142us) server_negotiation.cc:299] Negotiation successful
0621 05:20:42.431253 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260621 05:20:42.431633 5235 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51432
I20260621 05:20:42.432052 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 5297
I20260621 05:20:42.436465 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 5431
I20260621 05:20:42.441326 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 5565
I20260621 05:20:42.445952 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 5203
2026-06-21T05:20:42Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1947 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:42 dist-test-slave-b357 krb5kdc[5706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:42 dist-test-slave-b357 krb5kdc[5706](info): set up 2 sockets
Jun 21 05:20:42 dist-test-slave-b357 krb5kdc[5706](info): commencing operation
krb5kdc: starting...
W20260621 05:20:43.519315 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:20:43 dist-test-slave-b357 krb5kdc[5706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:20:43.642596 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:33057
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:43907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:33057
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:43.734608 5719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:43.734845 5719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:43.734879 5719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:43.736011 5719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:43.736053 5719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:43.736111 5719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:43.736132 5719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:43.736164 5719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:43.737628 5719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:43907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:33057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:33057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:43.737887 5719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:43.738107 5719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:20:43.743242 5719 server_base.cc:1034] running on GCE node
W20260621 05:20:43.743180 5724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:43.743213 5725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:43.743160 5727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:43.743882 5719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:43.744273 5719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:43.745597 5719 hybrid_clock.cc:648] HybridClock initialized: now 1782019243745566 us; error 46 us; skew 500 ppm
Jun 21 05:20:43 dist-test-slave-b357 krb5kdc[5706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019243, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:43.753332 5719 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:20:43.753862 5719 webserver.cc:460] Webserver started at http://127.2.32.62:36471/ using document root <none> and password file <none>
I20260621 05:20:43.754163 5719 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:43.754215 5719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:43.754408 5719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:43.755738 5719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "24567ac6b1ec4d43b70373c29fbdbb2e"
format_stamp: "Formatted at 2026-06-21 05:20:43 on dist-test-slave-b357"
server_key: "2f4728acbe990c9dc89bc6f8b25c3ec7"
server_key_iv: "7b7912661a69ecd60ea0db4eebc8aaba"
server_key_version: "encryptionkey@0"
I20260621 05:20:43.756116 5719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "24567ac6b1ec4d43b70373c29fbdbb2e"
format_stamp: "Formatted at 2026-06-21 05:20:43 on dist-test-slave-b357"
server_key: "2f4728acbe990c9dc89bc6f8b25c3ec7"
server_key_iv: "7b7912661a69ecd60ea0db4eebc8aaba"
server_key_version: "encryptionkey@0"
I20260621 05:20:43.757606 5719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:20:43.758829 5734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:43.759058 5719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:43.759202 5719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "24567ac6b1ec4d43b70373c29fbdbb2e"
format_stamp: "Formatted at 2026-06-21 05:20:43 on dist-test-slave-b357"
server_key: "2f4728acbe990c9dc89bc6f8b25c3ec7"
server_key_iv: "7b7912661a69ecd60ea0db4eebc8aaba"
server_key_version: "encryptionkey@0"
I20260621 05:20:43.759282 5719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:43.777726 5719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:43.779954 5719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:43.780099 5719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:43.784407 5719 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:33057
I20260621 05:20:43.784457 5786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:33057 every 8 connection(s)
I20260621 05:20:43.784950 5719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:43.785631 5787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:43.788566 5787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e: Bootstrap starting.
I20260621 05:20:43.789069 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 5719
I20260621 05:20:43.789224 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:20:43.789495 2176 external_mini_cluster.cc:1428] Setting key 056d028694b326b7e2b1ecd2987614ed
I20260621 05:20:43.789556 5787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:43.790045 5787 log.cc:824] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:43.791112 5787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e: No bootstrap required, opened a new log
I20260621 05:20:43.793448 5787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24567ac6b1ec4d43b70373c29fbdbb2e" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 33057 } }
I20260621 05:20:43.793610 5787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:43.793676 5787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24567ac6b1ec4d43b70373c29fbdbb2e, State: Initialized, Role: FOLLOWER
I20260621 05:20:43.793839 5787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24567ac6b1ec4d43b70373c29fbdbb2e" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 33057 } }
I20260621 05:20:43.793926 5787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:43.793982 5787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:43.794035 5787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:43.794747 5787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24567ac6b1ec4d43b70373c29fbdbb2e" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 33057 } }
I20260621 05:20:43.794883 5787 leader_election.cc:302] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24567ac6b1ec4d43b70373c29fbdbb2e; no voters:
I20260621 05:20:43.795164 5787 leader_election.cc:288] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:43.795190 5792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:43.795406 5792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e [term 1 LEADER]: Becoming Leader. State: Replica: 24567ac6b1ec4d43b70373c29fbdbb2e, State: Running, Role: LEADER
I20260621 05:20:43.795506 5787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:43.795619 5792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24567ac6b1ec4d43b70373c29fbdbb2e" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 33057 } }
I20260621 05:20:43.796113 5792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24567ac6b1ec4d43b70373c29fbdbb2e. Latest consensus state: current_term: 1 leader_uuid: "24567ac6b1ec4d43b70373c29fbdbb2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24567ac6b1ec4d43b70373c29fbdbb2e" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 33057 } } }
I20260621 05:20:43.796159 5794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24567ac6b1ec4d43b70373c29fbdbb2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24567ac6b1ec4d43b70373c29fbdbb2e" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 33057 } } }
I20260621 05:20:43.796321 5794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:43.796554 5792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:43.797178 5801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 21 05:20:43 dist-test-slave-b357 krb5kdc[5706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:43.797765 5801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:43.800159 5801 catalog_manager.cc:1347] Generated new cluster ID: 9d49c346285a43f991f3ac203afd6e14
I20260621 05:20:43.800215 5801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:43.802772 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:43.790206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33570 (local address 127.2.32.62:33057)
0621 05:20:43.790423 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:43.790435 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:43.790478 (+ 43us) server_negotiation.cc:407] Connection header received
0621 05:20:43.791091 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:43.791098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:43.791232 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:43.791458 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:43.792214 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:43.793277 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:43.794502 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:43.795057 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:43.795214 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:43.798278 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:43.798284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:43.798284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:43.798306 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:43.800518 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:43.801029 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:43.801034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:43.801035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:43.801144 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:43.801396 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:43.801397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:43.801398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:43.801741 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:43.801962 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:43.802352 (+ 390us) server_negotiation.cc:299] Negotiation successful
0621 05:20:43.802531 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":126,"threads_started":1}
I20260621 05:20:43.808964 5801 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:43.809726 5801 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:43.817251 5801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24567ac6b1ec4d43b70373c29fbdbb2e: Generated new TSK 0
I20260621 05:20:43.817449 5801 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
I20260621 05:20:43.847378 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:33057
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:43907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:43.932564 5813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:43.932803 5813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:43.932838 5813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:43.933935 5813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:43.934002 5813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:43.934047 5813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:20:43.935510 5813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:43907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:33057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:43.935761 5813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:43.935981 5813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:43.941807 5818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:43.941815 5821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:43.941812 5819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:43.942123 5813 server_base.cc:1034] running on GCE node
I20260621 05:20:43.942360 5813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:43.942607 5813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:43.943826 5813 hybrid_clock.cc:648] HybridClock initialized: now 1782019243943801 us; error 30 us; skew 500 ppm
Jun 21 05:20:43 dist-test-slave-b357 krb5kdc[5706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019243, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:43.947433 5813 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:20:43.948081 5813 webserver.cc:460] Webserver started at http://127.2.32.1:38965/ using document root <none> and password file <none>
I20260621 05:20:43.948346 5813 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:43.948400 5813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:43.948508 5813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:43.949787 5813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "e6ebdd8d40624829a8a5267832f2c0d5"
format_stamp: "Formatted at 2026-06-21 05:20:43 on dist-test-slave-b357"
server_key: "37d738c3cf477dc56feb590b71721831"
server_key_iv: "72b7d725b85d5c8de1a94e98b9eb057b"
server_key_version: "encryptionkey@0"
I20260621 05:20:43.950199 5813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "e6ebdd8d40624829a8a5267832f2c0d5"
format_stamp: "Formatted at 2026-06-21 05:20:43 on dist-test-slave-b357"
server_key: "37d738c3cf477dc56feb590b71721831"
server_key_iv: "72b7d725b85d5c8de1a94e98b9eb057b"
server_key_version: "encryptionkey@0"
I20260621 05:20:43.951875 5813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:20:43.952909 5828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:43.953167 5813 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:43.953285 5813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "e6ebdd8d40624829a8a5267832f2c0d5"
format_stamp: "Formatted at 2026-06-21 05:20:43 on dist-test-slave-b357"
server_key: "37d738c3cf477dc56feb590b71721831"
server_key_iv: "72b7d725b85d5c8de1a94e98b9eb057b"
server_key_version: "encryptionkey@0"
I20260621 05:20:43.953413 5813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:43.964224 5813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:43.966457 5813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:43.966566 5813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:43.966737 5813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:43.967049 5813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:43.967082 5813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:43.967162 5813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:43.967188 5813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:43.973119 5813 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:35707
I20260621 05:20:43.973311 5941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:35707 every 8 connection(s)
I20260621 05:20:43.973523 5813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:20:43.973825 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 5813
I20260621 05:20:43.973939 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:20:43.974143 2176 external_mini_cluster.cc:1428] Setting key 1dfd12e9e56d57ef45c173215b58321b
Jun 21 05:20:43 dist-test-slave-b357 krb5kdc[5706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019243, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.32.2@KRBTEST.COM; defaulting to no policy
I20260621 05:20:43.984102 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:43.974344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:55797 (local address 127.2.32.62:33057)
0621 05:20:43.974390 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:43.974392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:43.974632 (+ 240us) server_negotiation.cc:407] Connection header received
0621 05:20:43.975711 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:43.975712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:43.975724 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:43.975779 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:43.976385 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:43.976899 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:43.978473 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:43.978688 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:43.978762 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:43.981201 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:43.981202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:43.981203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:43.981211 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:43.982678 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:43.983096 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:43.983097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:43.983098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:43.983200 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:43.983457 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:43.983458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:43.983458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:43.983594 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:43.983673 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:43.983931 (+ 258us) server_negotiation.cc:299] Negotiation successful
0621 05:20:43.984009 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:20:43.984318 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:43.974334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:33057 (local address 127.2.32.1:55797)
0621 05:20:43.974493 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:43.974543 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:43.975606 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:43.975849 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:43.975853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:43.976101 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:43.976287 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:43.976291 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:43.976998 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:43.976999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:43.978339 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:43.978342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:43.978783 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:43.978784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:43.978929 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:43.979478 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:43.979488 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:43.981087 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:43.982782 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:43.982788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:43.982790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:43.983008 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:43.983274 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:43.983275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:43.983276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:43.983385 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:43.983664 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:43.983670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:43.983804 (+ 134us) client_negotiation.cc:769] Sending connection context
0621 05:20:43.983911 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 05:20:43.984059 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260621 05:20:43.984719 5942 heartbeater.cc:347] Connected to a master server at 127.2.32.62:33057
I20260621 05:20:43.984803 5942 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.2@KRBTEST.COM" created.
I20260621 05:20:43.985013 5942 heartbeater.cc:511] Master 127.2.32.62:33057 requested a full tablet report, sending...
I20260621 05:20:43.985596 5751 ts_manager.cc:194] Registered new tserver with Master: e6ebdd8d40624829a8a5267832f2c0d5 (127.2.32.1:35707)
I20260621 05:20:43.986513 5751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:55797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
I20260621 05:20:44.003218 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:33057
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:43907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:44.093025 5947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:44.093333 5947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:44.093381 5947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:44.094779 5947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:44.094856 5947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:44.094924 5947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:20:44.096916 5947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:43907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:33057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:44.097237 5947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:44.097514 5947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:44.103067 5955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:44.103067 5952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:44.103240 5953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:44.103366 5947 server_base.cc:1034] running on GCE node
I20260621 05:20:44.103605 5947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:44.103875 5947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:44.105050 5947 hybrid_clock.cc:648] HybridClock initialized: now 1782019244105022 us; error 57 us; skew 500 ppm
Jun 21 05:20:44 dist-test-slave-b357 krb5kdc[5706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019244, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:44.108239 5947 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:20:44.108917 5947 webserver.cc:460] Webserver started at http://127.2.32.2:34621/ using document root <none> and password file <none>
I20260621 05:20:44.109174 5947 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:44.109223 5947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:44.109360 5947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:44.110756 5947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "e4d7b8aa64ef4c0485e659095912b7a6"
format_stamp: "Formatted at 2026-06-21 05:20:44 on dist-test-slave-b357"
server_key: "a7da98c85f9a72da9129599844a28d93"
server_key_iv: "10a465bbb2738f5aa0f432d21cd62b4c"
server_key_version: "encryptionkey@0"
I20260621 05:20:44.111186 5947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "e4d7b8aa64ef4c0485e659095912b7a6"
format_stamp: "Formatted at 2026-06-21 05:20:44 on dist-test-slave-b357"
server_key: "a7da98c85f9a72da9129599844a28d93"
server_key_iv: "10a465bbb2738f5aa0f432d21cd62b4c"
server_key_version: "encryptionkey@0"
I20260621 05:20:44.112895 5947 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260621 05:20:44.113842 5962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:44.114166 5947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260621 05:20:44.114281 5947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "e4d7b8aa64ef4c0485e659095912b7a6"
format_stamp: "Formatted at 2026-06-21 05:20:44 on dist-test-slave-b357"
server_key: "a7da98c85f9a72da9129599844a28d93"
server_key_iv: "10a465bbb2738f5aa0f432d21cd62b4c"
server_key_version: "encryptionkey@0"
I20260621 05:20:44.114384 5947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:44.134042 5947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:44.136600 5947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:44.136786 5947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:44.137022 5947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:44.137573 5947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:44.137636 5947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:44.137674 5947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:44.137702 5947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:44.143579 5947 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:41919
I20260621 05:20:44.143689 6075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:41919 every 8 connection(s)
I20260621 05:20:44.143966 5947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:20:44.149641 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 5947
I20260621 05:20:44.149962 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:20:44.150203 2176 external_mini_cluster.cc:1428] Setting key 8df0b2e275b058f0bb0373b26e88a7b9
Jun 21 05:20:44 dist-test-slave-b357 krb5kdc[5706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019244, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:44.155452 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:44.144708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:46431 (local address 127.2.32.62:33057)
0621 05:20:44.144783 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:44.144785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:44.144948 (+ 163us) server_negotiation.cc:407] Connection header received
0621 05:20:44.145958 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:44.145960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:44.145976 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:44.146069 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:44.146738 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:44.147431 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:44.148815 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:44.149054 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:44.149138 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:44.151997 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:44.151999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:44.152000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:44.152018 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:44.153684 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:44.154170 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:44.154171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:44.154171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:44.154229 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:44.154511 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:44.154511 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:44.154512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:44.154650 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:44.154705 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:44.155126 (+ 421us) server_negotiation.cc:299] Negotiation successful
0621 05:20:44.155308 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 05:20:44.155514 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:44.144661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:33057 (local address 127.2.32.2:46431)
0621 05:20:44.144832 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:44.144866 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:44.145758 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:44.146154 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:44.146157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:44.146408 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:44.146601 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:44.146605 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:44.147582 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:44.147584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:44.148702 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:44.148704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:44.149179 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:44.149180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:44.149333 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:44.150224 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:44.150234 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:44.151878 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:44.153766 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:44.153771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:44.153774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:44.154080 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:44.154300 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:44.154301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:44.154301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:44.154437 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:44.154716 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:44.154721 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:44.154890 (+ 169us) client_negotiation.cc:769] Sending connection context
0621 05:20:44.155007 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 05:20:44.155237 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
I20260621 05:20:44.155902 6076 heartbeater.cc:347] Connected to a master server at 127.2.32.62:33057
I20260621 05:20:44.155998 6076 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:44.156194 6076 heartbeater.cc:511] Master 127.2.32.62:33057 requested a full tablet report, sending...
I20260621 05:20:44.156781 5751 ts_manager.cc:194] Registered new tserver with Master: e4d7b8aa64ef4c0485e659095912b7a6 (127.2.32.2:41919)
I20260621 05:20:44.157370 5751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:46431
WARNING: no policy specified for kudu/127.2.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
I20260621 05:20:44.178903 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:33057
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:43907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:44.260754 6081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:44.260979 6081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:44.261018 6081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:44.262297 6081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:44.262354 6081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:44.262398 6081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:20:44.264524 6081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:43907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:33057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:44.264791 6081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:44.265020 6081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:44.271054 6087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:44.271054 6089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:44.271144 6086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:44.271257 6081 server_base.cc:1034] running on GCE node
I20260621 05:20:44.271597 6081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:44.271862 6081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:44.273116 6081 hybrid_clock.cc:648] HybridClock initialized: now 1782019244273080 us; error 81 us; skew 500 ppm
Jun 21 05:20:44 dist-test-slave-b357 krb5kdc[5706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019244, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:44.275929 6081 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:20:44.276400 6081 webserver.cc:460] Webserver started at http://127.2.32.3:44249/ using document root <none> and password file <none>
I20260621 05:20:44.276587 6081 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:44.276628 6081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:44.276718 6081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:44.277675 6081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "e6196d4619e341e6ac10e62233157f84"
format_stamp: "Formatted at 2026-06-21 05:20:44 on dist-test-slave-b357"
server_key: "e1db5285a682826e68a2bc6e92e7f8f4"
server_key_iv: "66a0aad9e0333d97871d9ef68d66c376"
server_key_version: "encryptionkey@0"
I20260621 05:20:44.277971 6081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "e6196d4619e341e6ac10e62233157f84"
format_stamp: "Formatted at 2026-06-21 05:20:44 on dist-test-slave-b357"
server_key: "e1db5285a682826e68a2bc6e92e7f8f4"
server_key_iv: "66a0aad9e0333d97871d9ef68d66c376"
server_key_version: "encryptionkey@0"
I20260621 05:20:44.279291 6081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:20:44.280085 6096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:44.280378 6081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:44.280459 6081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "e6196d4619e341e6ac10e62233157f84"
format_stamp: "Formatted at 2026-06-21 05:20:44 on dist-test-slave-b357"
server_key: "e1db5285a682826e68a2bc6e92e7f8f4"
server_key_iv: "66a0aad9e0333d97871d9ef68d66c376"
server_key_version: "encryptionkey@0"
I20260621 05:20:44.280517 6081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:44.298904 6081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:44.300890 6081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:44.300998 6081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:44.301160 6081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:44.301450 6081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:44.301481 6081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:44.301504 6081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:44.301553 6081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:44.307237 6081 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:44445
I20260621 05:20:44.307310 6209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:44445 every 8 connection(s)
I20260621 05:20:44.307615 6081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:20:44 dist-test-slave-b357 krb5kdc[5706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019244, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:44.314827 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 6081
I20260621 05:20:44.314947 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:20:44.315176 2176 external_mini_cluster.cc:1428] Setting key cbf178af8ca8a84442889644b8cdd2de
I20260621 05:20:44.318662 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:44.308655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:56561 (local address 127.2.32.62:33057)
0621 05:20:44.308729 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:44.308731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:44.309000 (+ 269us) server_negotiation.cc:407] Connection header received
0621 05:20:44.309984 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:44.309985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:44.309998 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:44.310055 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:44.310643 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:44.311225 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:44.312186 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:44.312386 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:44.312471 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:44.315085 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:44.315087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:44.315087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:44.315096 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:44.316903 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:44.317534 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:44.317535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:44.317535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:44.317608 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:44.317931 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:44.317932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:44.317933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:44.318075 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:44.318152 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:44.318500 (+ 348us) server_negotiation.cc:299] Negotiation successful
0621 05:20:44.318584 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 05:20:44.318918 6212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:44.308623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:33057 (local address 127.2.32.3:56561)
0621 05:20:44.308864 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:44.308901 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:44.309872 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:44.310136 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:44.310139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:44.310373 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:44.310551 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:44.310555 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:44.311300 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:44.311301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:44.312087 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:44.312089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:44.312479 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:44.312480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:44.312610 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:44.313276 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:44.313283 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:44.314887 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:44.317022 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:44.317033 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:44.317036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:44.317444 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:44.317709 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:44.317711 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:44.317711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:44.317838 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:44.318158 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:44.318163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:44.318373 (+ 210us) client_negotiation.cc:769] Sending connection context
0621 05:20:44.318478 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 05:20:44.318624 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":133,"threads_started":1}
I20260621 05:20:44.319440 6210 heartbeater.cc:347] Connected to a master server at 127.2.32.62:33057
I20260621 05:20:44.319548 6210 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:44.319741 6210 heartbeater.cc:511] Master 127.2.32.62:33057 requested a full tablet report, sending...
I20260621 05:20:44.320219 5751 ts_manager.cc:194] Registered new tserver with Master: e6196d4619e341e6ac10e62233157f84 (127.2.32.3:44445)
I20260621 05:20:44.320554 5751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:56561
I20260621 05:20:44.327977 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 05:20:44 dist-test-slave-b357 krb5kdc[5706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 05:20:44 dist-test-slave-b357 krb5kdc[5706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.1@KRBTEST.COM
I20260621 05:20:44.355156 6217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:44.345552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.2.32.1:35707)
0621 05:20:44.345801 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:44.345806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:44.345819 (+ 13us) server_negotiation.cc:407] Connection header received
0621 05:20:44.345846 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:44.345849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:44.345863 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:44.345948 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:44.346700 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:44.347733 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:44.348595 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:44.349056 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:44.349131 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:44.351862 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:44.351876 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:44.351876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:44.351889 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:44.353712 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:44.354105 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:44.354108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:44.354108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:44.354167 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:44.354400 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:44.354401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:44.354402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:44.354539 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:44.354712 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:44.354759 (+ 47us) server_negotiation.cc:299] Negotiation successful
0621 05:20:44.354828 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":148,"threads_started":1}
Jun 21 05:20:44 dist-test-slave-b357 krb5kdc[5706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:44.363950 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:44.356524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33584 (local address 127.2.32.62:33057)
0621 05:20:44.356614 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:44.356616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:44.356652 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:20:44.356720 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:44.356720 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:44.356729 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:44.356774 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:44.357573 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:44.358056 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:44.358882 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:44.359094 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:44.359251 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:44.361402 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:44.361405 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:44.361406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:44.361417 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:44.362758 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:44.363148 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:44.363150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:44.363150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:44.363206 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:44.363417 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:44.363418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:44.363418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:44.363559 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:44.363672 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:44.363773 (+ 101us) server_negotiation.cc:299] Negotiation successful
0621 05:20:44.363846 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260621 05:20:44.364435 5751 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33584
I20260621 05:20:44.364995 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 5813
I20260621 05:20:44.369968 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 5947
I20260621 05:20:44.374413 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 6081
I20260621 05:20:44.379252 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 5719
2026-06-21T05:20:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1932 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:44 dist-test-slave-b357 krb5kdc[6222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:44 dist-test-slave-b357 krb5kdc[6222](info): set up 2 sockets
Jun 21 05:20:44 dist-test-slave-b357 krb5kdc[6222](info): commencing operation
krb5kdc: starting...
W20260621 05:20:45.443584 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:20:45 dist-test-slave-b357 krb5kdc[6222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:20:45.566143 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:36493
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:46695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:36493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:45.650738 6235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:45.650933 6235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:45.650955 6235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:45.652134 6235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:45.652182 6235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:45.652197 6235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:45.652209 6235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:45.652221 6235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:45.653676 6235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:46695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:36493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:36493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:45.653905 6235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:45.654337 6235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:45.659489 6240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:45.659489 6241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:45.659580 6243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:45.660104 6235 server_base.cc:1034] running on GCE node
I20260621 05:20:45.660364 6235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:45.660809 6235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:45.662065 6235 hybrid_clock.cc:648] HybridClock initialized: now 1782019245662032 us; error 47 us; skew 500 ppm
Jun 21 05:20:45 dist-test-slave-b357 krb5kdc[6222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019245, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:45.665124 6235 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:20:45.665547 6235 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:45.665619 6235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:45.665836 6235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:45.666942 6235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "323e46ba44ea45a8839f1a3164619a70"
format_stamp: "Formatted at 2026-06-21 05:20:45 on dist-test-slave-b357"
server_key: "2958c99e849a22cb8298c45e54d7fe9c"
server_key_iv: "d47ffa25064c331fcbb0b1f8529e2948"
server_key_version: "encryptionkey@0"
I20260621 05:20:45.667357 6235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "323e46ba44ea45a8839f1a3164619a70"
format_stamp: "Formatted at 2026-06-21 05:20:45 on dist-test-slave-b357"
server_key: "2958c99e849a22cb8298c45e54d7fe9c"
server_key_iv: "d47ffa25064c331fcbb0b1f8529e2948"
server_key_version: "encryptionkey@0"
I20260621 05:20:45.669067 6235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:45.670188 6249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:45.670555 6235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:45.670701 6235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "323e46ba44ea45a8839f1a3164619a70"
format_stamp: "Formatted at 2026-06-21 05:20:45 on dist-test-slave-b357"
server_key: "2958c99e849a22cb8298c45e54d7fe9c"
server_key_iv: "d47ffa25064c331fcbb0b1f8529e2948"
server_key_version: "encryptionkey@0"
I20260621 05:20:45.670804 6235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:45.692658 6235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:45.695142 6235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:45.695317 6235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:45.699556 6235 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:36493
I20260621 05:20:45.699581 6301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:36493 every 8 connection(s)
I20260621 05:20:45.699899 6235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:45.700834 6302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:45.701959 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 6235
I20260621 05:20:45.702106 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:20:45.702302 2176 external_mini_cluster.cc:1428] Setting key 0372e3b4aeb008e1a8b2ee747efdd4b6
I20260621 05:20:45.703610 6302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70: Bootstrap starting.
I20260621 05:20:45.704337 6302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:45.704646 6302 log.cc:824] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:45.705330 6302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70: No bootstrap required, opened a new log
I20260621 05:20:45.707048 6302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323e46ba44ea45a8839f1a3164619a70" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36493 } }
I20260621 05:20:45.707274 6302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:45.707324 6302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 323e46ba44ea45a8839f1a3164619a70, State: Initialized, Role: FOLLOWER
I20260621 05:20:45.707449 6302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323e46ba44ea45a8839f1a3164619a70" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36493 } }
I20260621 05:20:45.707545 6302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:45.707587 6302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:45.707636 6302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:45.708348 6302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323e46ba44ea45a8839f1a3164619a70" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36493 } }
I20260621 05:20:45.708470 6302 leader_election.cc:302] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 323e46ba44ea45a8839f1a3164619a70; no voters:
I20260621 05:20:45.708626 6302 leader_election.cc:288] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:45.708752 6307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:45.708957 6302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:45.709007 6307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70 [term 1 LEADER]: Becoming Leader. State: Replica: 323e46ba44ea45a8839f1a3164619a70, State: Running, Role: LEADER
I20260621 05:20:45.709177 6307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323e46ba44ea45a8839f1a3164619a70" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36493 } }
I20260621 05:20:45.709733 6308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "323e46ba44ea45a8839f1a3164619a70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323e46ba44ea45a8839f1a3164619a70" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36493 } } }
I20260621 05:20:45.709851 6308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:45.709745 6309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 323e46ba44ea45a8839f1a3164619a70. Latest consensus state: current_term: 1 leader_uuid: "323e46ba44ea45a8839f1a3164619a70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323e46ba44ea45a8839f1a3164619a70" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36493 } } }
I20260621 05:20:45.710145 6316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 21 05:20:45 dist-test-slave-b357 krb5kdc[6222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:45.710163 6309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:45.710858 6316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:45.713972 6316 catalog_manager.cc:1347] Generated new cluster ID: 4d750c186e6d4178a7516b4d660a4354
I20260621 05:20:45.714037 6316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:45.715320 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:45.703031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47124 (local address 127.2.32.62:36493)
0621 05:20:45.703286 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:45.703296 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:45.703333 (+ 37us) server_negotiation.cc:407] Connection header received
0621 05:20:45.703850 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:45.703859 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:45.703920 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:45.704103 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:45.705057 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:45.706137 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:45.706948 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:45.707580 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:45.707703 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:45.710853 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:45.710859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:45.710859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:45.710873 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:45.713490 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:45.713981 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:45.713984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:45.713984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:45.714107 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:45.714336 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:45.714337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:45.714337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:45.714605 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:45.714777 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:45.714896 (+ 119us) server_negotiation.cc:299] Negotiation successful
0621 05:20:45.715054 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":159,"threads_started":1}
I20260621 05:20:45.741321 6316 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:45.742056 6316 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:45.746043 6316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70: Generated new TSK 0
I20260621 05:20:45.746315 6316 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
I20260621 05:20:45.786120 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:36493
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:46695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:45.878043 6328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:45.878242 6328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:45.878268 6328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:45.879407 6328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:45.879458 6328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:45.879488 6328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:20:45.880910 6328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:46695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:36493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:45.881084 6328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:45.881248 6328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:45.886502 6336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:45.886515 6333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:45.886512 6334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:45.886755 6328 server_base.cc:1034] running on GCE node
I20260621 05:20:45.887200 6328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:45.887498 6328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:45.888671 6328 hybrid_clock.cc:648] HybridClock initialized: now 1782019245888666 us; error 39 us; skew 500 ppm
Jun 21 05:20:45 dist-test-slave-b357 krb5kdc[6222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019245, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:45.891911 6328 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:20:45.892443 6328 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:45.892505 6328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:45.892632 6328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:45.894037 6328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "3cf785c338ba4e38abdf761425728097"
format_stamp: "Formatted at 2026-06-21 05:20:45 on dist-test-slave-b357"
server_key: "2a97665df6ace0c82fc385f937574a8e"
server_key_iv: "752ccfa27e8171442e1b232aca5355a6"
server_key_version: "encryptionkey@0"
I20260621 05:20:45.894475 6328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "3cf785c338ba4e38abdf761425728097"
format_stamp: "Formatted at 2026-06-21 05:20:45 on dist-test-slave-b357"
server_key: "2a97665df6ace0c82fc385f937574a8e"
server_key_iv: "752ccfa27e8171442e1b232aca5355a6"
server_key_version: "encryptionkey@0"
I20260621 05:20:45.896122 6328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260621 05:20:45.897217 6342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:45.897596 6328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:45.897696 6328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "3cf785c338ba4e38abdf761425728097"
format_stamp: "Formatted at 2026-06-21 05:20:45 on dist-test-slave-b357"
server_key: "2a97665df6ace0c82fc385f937574a8e"
server_key_iv: "752ccfa27e8171442e1b232aca5355a6"
server_key_version: "encryptionkey@0"
I20260621 05:20:45.897827 6328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:45.916924 6328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:45.919543 6328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:45.919696 6328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:45.920051 6328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:45.920444 6328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:45.920480 6328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:45.920503 6328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:45.920516 6328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:45.926488 6328 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:39271
I20260621 05:20:45.926522 6455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:39271 every 8 connection(s)
I20260621 05:20:45.926790 6328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:20:45.932984 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 6328
I20260621 05:20:45.933118 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:20:45.933341 2176 external_mini_cluster.cc:1428] Setting key 00bd4c77dc86cae205e9afd31d7d60a4
Jun 21 05:20:45 dist-test-slave-b357 krb5kdc[6222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019245, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:45.938386 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:45.927853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:36149 (local address 127.2.32.62:36493)
0621 05:20:45.927916 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:45.927920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:45.928001 (+ 81us) server_negotiation.cc:407] Connection header received
0621 05:20:45.928661 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:45.928664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:45.928678 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:45.928755 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:45.930116 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:45.930677 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:45.932112 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:45.932349 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:45.932421 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:45.934856 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:45.934864 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:45.934865 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:45.934872 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:45.936529 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:45.937144 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:45.937145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:45.937146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:45.937211 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:45.937606 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:45.937607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:45.937608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:45.937764 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:45.937842 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:45.938137 (+ 295us) server_negotiation.cc:299] Negotiation successful
0621 05:20:45.938247 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 05:20:45.938400 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:45.927618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:36493 (local address 127.2.32.1:36149)
0621 05:20:45.927855 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:45.927906 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:45.928513 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:45.928907 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:45.928912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:45.929301 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:45.929972 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:45.929979 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:45.930890 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:45.930891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:45.932002 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:45.932005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:45.932485 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:45.932486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:45.932629 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:45.933249 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:45.933257 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:45.934735 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:45.936659 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:45.936669 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:45.936672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:45.937038 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:45.937358 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:45.937360 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:45.937361 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:45.937520 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:45.937850 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:45.937859 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:45.937997 (+ 138us) client_negotiation.cc:769] Sending connection context
0621 05:20:45.938092 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 05:20:45.938239 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":144,"threads_started":1}
I20260621 05:20:45.938973 6456 heartbeater.cc:347] Connected to a master server at 127.2.32.62:36493
I20260621 05:20:45.939065 6456 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:45.939239 6456 heartbeater.cc:511] Master 127.2.32.62:36493 requested a full tablet report, sending...
I20260621 05:20:45.939867 6266 ts_manager.cc:194] Registered new tserver with Master: 3cf785c338ba4e38abdf761425728097 (127.2.32.1:39271)
I20260621 05:20:45.940483 6266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:36149
WARNING: no policy specified for kudu/127.2.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
I20260621 05:20:45.962762 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:36493
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:46695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:46.047437 6461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:46.047679 6461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:46.047719 6461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:46.048831 6461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:46.048899 6461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:46.048944 6461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:20:46.050410 6461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:46695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:36493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:46.050664 6461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:46.050882 6461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:46.055931 6469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:46.055902 6467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:46.055902 6466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:46.055958 6461 server_base.cc:1034] running on GCE node
I20260621 05:20:46.056493 6461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:46.056808 6461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:46.058003 6461 hybrid_clock.cc:648] HybridClock initialized: now 1782019246057942 us; error 77 us; skew 500 ppm
Jun 21 05:20:46 dist-test-slave-b357 krb5kdc[6222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019246, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:46.061113 6461 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:20:46.061477 6461 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:46.061529 6461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:46.061678 6461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:46.062714 6461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "c7cea9461e3644bc9bec463c389a561d"
format_stamp: "Formatted at 2026-06-21 05:20:46 on dist-test-slave-b357"
server_key: "045b77b52049684015e9fd6ca286f285"
server_key_iv: "1c281de5adbe667ab0554b6a010e3791"
server_key_version: "encryptionkey@0"
I20260621 05:20:46.063098 6461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "c7cea9461e3644bc9bec463c389a561d"
format_stamp: "Formatted at 2026-06-21 05:20:46 on dist-test-slave-b357"
server_key: "045b77b52049684015e9fd6ca286f285"
server_key_iv: "1c281de5adbe667ab0554b6a010e3791"
server_key_version: "encryptionkey@0"
I20260621 05:20:46.064503 6461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:46.065411 6475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:46.065699 6461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:46.065814 6461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "c7cea9461e3644bc9bec463c389a561d"
format_stamp: "Formatted at 2026-06-21 05:20:46 on dist-test-slave-b357"
server_key: "045b77b52049684015e9fd6ca286f285"
server_key_iv: "1c281de5adbe667ab0554b6a010e3791"
server_key_version: "encryptionkey@0"
I20260621 05:20:46.065896 6461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:46.089974 6461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:46.092427 6461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:46.092603 6461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:46.092844 6461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:46.093191 6461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:46.093252 6461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:46.093297 6461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:46.093329 6461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:46.099615 6461 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:37829
I20260621 05:20:46.099685 6588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:37829 every 8 connection(s)
I20260621 05:20:46.099974 6461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:20:46 dist-test-slave-b357 krb5kdc[6222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019246, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:46.108814 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 6461
I20260621 05:20:46.108952 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:20:46.109164 2176 external_mini_cluster.cc:1428] Setting key 2e715d9f0a63426a3fc3d74688acd8af
I20260621 05:20:46.111652 6591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.100713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:36493 (local address 127.2.32.2:55927)
0621 05:20:46.100870 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:46.100921 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:46.101414 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:46.101828 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:46.101831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:46.102243 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:46.102718 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:46.102724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.103633 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.103635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:46.104588 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:46.104593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.105038 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.105039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:46.105189 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:46.105880 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:46.105890 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:46.107524 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:46.109907 (+ 2383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:46.109914 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:46.109916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:46.110272 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:46.110539 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:46.110540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:46.110540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:46.110708 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:46.111090 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:46.111094 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:46.111280 (+ 186us) client_negotiation.cc:769] Sending connection context
0621 05:20:46.111369 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 05:20:46.111496 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
I20260621 05:20:46.111716 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.100715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:55927 (local address 127.2.32.62:36493)
0621 05:20:46.100758 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:46.100759 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:46.101047 (+ 288us) server_negotiation.cc:407] Connection header received
0621 05:20:46.101642 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:46.101644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:46.101666 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:46.101771 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:46.102864 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.103473 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.104705 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.104901 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.104974 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:46.107647 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:46.107648 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:46.107648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:46.107655 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:46.109716 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:46.110361 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:46.110363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:46.110363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:46.110444 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:46.110858 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:46.110859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:46.110860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:46.111011 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:46.111050 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:46.111430 (+ 380us) server_negotiation.cc:299] Negotiation successful
0621 05:20:46.111558 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:20:46.112334 6589 heartbeater.cc:347] Connected to a master server at 127.2.32.62:36493
I20260621 05:20:46.112447 6589 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:46.112596 6589 heartbeater.cc:511] Master 127.2.32.62:36493 requested a full tablet report, sending...
I20260621 05:20:46.113077 6266 ts_manager.cc:194] Registered new tserver with Master: c7cea9461e3644bc9bec463c389a561d (127.2.32.2:37829)
I20260621 05:20:46.113622 6266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:55927
WARNING: no policy specified for kudu/127.2.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
I20260621 05:20:46.138379 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:36493
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:46695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:46.224886 6594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:46.225176 6594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:46.225212 6594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:46.227001 6594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:46.227085 6594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:46.227200 6594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:20:46.228816 6594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:46695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:36493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:46.229079 6594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:46.229310 6594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:46.234390 6599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:46.234529 6602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:46.234352 6600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:46.234625 6594 server_base.cc:1034] running on GCE node
I20260621 05:20:46.234813 6594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:46.235080 6594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:46.236243 6594 hybrid_clock.cc:648] HybridClock initialized: now 1782019246236219 us; error 43 us; skew 500 ppm
Jun 21 05:20:46 dist-test-slave-b357 krb5kdc[6222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019246, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:46.239391 6594 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:20:46.239779 6594 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:46.239852 6594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:46.239996 6594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:46.241017 6594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "cf3325a2ff2d4659a0deb2f4cbab0e31"
format_stamp: "Formatted at 2026-06-21 05:20:46 on dist-test-slave-b357"
server_key: "e7290cad0b8b044147552dd0cd3ebecf"
server_key_iv: "056160c0df59b6afb9ed2798db2330cf"
server_key_version: "encryptionkey@0"
I20260621 05:20:46.241393 6594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "cf3325a2ff2d4659a0deb2f4cbab0e31"
format_stamp: "Formatted at 2026-06-21 05:20:46 on dist-test-slave-b357"
server_key: "e7290cad0b8b044147552dd0cd3ebecf"
server_key_iv: "056160c0df59b6afb9ed2798db2330cf"
server_key_version: "encryptionkey@0"
I20260621 05:20:46.242794 6594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:20:46.243891 6608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:46.244277 6594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:46.244410 6594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "cf3325a2ff2d4659a0deb2f4cbab0e31"
format_stamp: "Formatted at 2026-06-21 05:20:46 on dist-test-slave-b357"
server_key: "e7290cad0b8b044147552dd0cd3ebecf"
server_key_iv: "056160c0df59b6afb9ed2798db2330cf"
server_key_version: "encryptionkey@0"
I20260621 05:20:46.244504 6594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:46.262568 6594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:46.265568 6594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:46.265717 6594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:46.266063 6594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:46.266443 6594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:46.266479 6594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:46.266503 6594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:46.266516 6594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:46.274612 6594 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:41253
I20260621 05:20:46.274709 6721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:41253 every 8 connection(s)
I20260621 05:20:46.275063 6594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:20:46 dist-test-slave-b357 krb5kdc[6222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019246, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:46.285017 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 6594
I20260621 05:20:46.285141 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:20:46.285353 2176 external_mini_cluster.cc:1428] Setting key cd03268721a12e6b6d7f07fae71494e5
I20260621 05:20:46.285748 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.275880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:45069 (local address 127.2.32.62:36493)
0621 05:20:46.275958 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:46.275960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:46.276127 (+ 167us) server_negotiation.cc:407] Connection header received
0621 05:20:46.276609 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:46.276610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:46.276622 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:46.276699 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:46.277625 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.278269 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.279362 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.279571 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.279638 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:46.281840 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:46.281842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:46.281842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:46.281854 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:46.283631 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:46.284323 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:46.284324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:46.284324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:46.284381 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:46.284893 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:46.284894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:46.284894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:46.285022 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:46.285083 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:46.285470 (+ 387us) server_negotiation.cc:299] Negotiation successful
0621 05:20:46.285561 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 05:20:46.285864 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.275827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:36493 (local address 127.2.32.3:45069)
0621 05:20:46.276002 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:46.276042 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:46.276504 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:46.276750 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:46.276753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:46.277000 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:46.277486 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:46.277490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.278394 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.278394 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:46.279253 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:46.279255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.279652 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.279652 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:46.279751 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:46.280260 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:46.280270 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:46.281711 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:46.283858 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:46.283873 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:46.283875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:46.284219 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:46.284637 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:46.284639 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:46.284639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:46.284762 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:46.285104 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:46.285112 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:46.285281 (+ 169us) client_negotiation.cc:769] Sending connection context
0621 05:20:46.285413 (+ 132us) client_negotiation.cc:240] Negotiation successful
0621 05:20:46.285621 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260621 05:20:46.286324 6722 heartbeater.cc:347] Connected to a master server at 127.2.32.62:36493
I20260621 05:20:46.286422 6722 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:46.286532 6722 heartbeater.cc:511] Master 127.2.32.62:36493 requested a full tablet report, sending...
I20260621 05:20:46.286908 6266 ts_manager.cc:194] Registered new tserver with Master: cf3325a2ff2d4659a0deb2f4cbab0e31 (127.2.32.3:41253)
I20260621 05:20:46.287297 6266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:45069
I20260621 05:20:46.297344 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:20:46.305670 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.298660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47132 (local address 127.2.32.62:36493)
0621 05:20:46.298759 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:46.298761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:46.298778 (+ 17us) server_negotiation.cc:407] Connection header received
0621 05:20:46.298836 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:46.298837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:46.298845 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:46.298900 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:46.299563 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.300040 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.300870 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.301083 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.301139 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:46.302454 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:46.302455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:46.302455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:46.302461 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:46.304026 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:46.304517 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:46.304519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:46.304519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:46.304588 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:46.304937 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:46.304939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:46.304939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:46.305191 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:46.305316 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:46.305479 (+ 163us) server_negotiation.cc:299] Negotiation successful
0621 05:20:46.305569 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 05:20:46.307711 6266 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:20:46.309864 6266 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:20:46.317303 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.313153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:41253 (local address 127.0.0.1:58574)
0621 05:20:46.313277 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:46.313317 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:46.313375 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:46.313622 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:46.313625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:46.313627 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:46.313698 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:46.313701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.314828 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.314828 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:46.316263 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:46.316266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.316975 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.316976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:46.317055 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:46.317068 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 05:20:46.317165 (+ 97us) client_negotiation.cc:240] Negotiation successful
0621 05:20:46.317223 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":35,"thread_start_us":76,"threads_started":1}
I20260621 05:20:46.317335 6735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.313272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58574 (local address 127.2.32.3:41253)
0621 05:20:46.313450 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:46.313454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:46.313464 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:20:46.313500 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:46.313502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:46.313518 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:46.313585 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:46.314022 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.314704 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.316373 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.316860 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.316965 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:46.317025 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:46.317190 (+ 165us) server_negotiation.cc:299] Negotiation successful
0621 05:20:46.317235 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":82,"threads_started":1}
I20260621 05:20:46.317492 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.313542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37544 (local address 127.2.32.1:39271)
0621 05:20:46.313701 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:46.313704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:46.313711 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:20:46.313739 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:46.313740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:46.313779 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:46.313853 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:46.314062 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.314728 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.316351 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.317041 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.317115 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:46.317175 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:46.317363 (+ 188us) server_negotiation.cc:299] Negotiation successful
0621 05:20:46.317416 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260621 05:20:46.317727 6734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.313161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:39271 (local address 127.0.0.1:37544)
0621 05:20:46.313329 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:46.313340 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:46.313392 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:46.313938 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:46.313939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:46.313941 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:46.313983 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:46.313985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.314826 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.314827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:46.316270 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:46.316272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.317203 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.317204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:46.317260 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:46.317263 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:20:46.317627 (+ 364us) client_negotiation.cc:240] Negotiation successful
0621 05:20:46.317672 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":75,"thread_start_us":102,"threads_started":1}
I20260621 05:20:46.318198 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.313374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:37829 (local address 127.0.0.1:40538)
0621 05:20:46.313807 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:46.313817 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:46.313861 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:46.315008 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:46.315009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:46.315012 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:46.315061 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:46.315063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.316008 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.316009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:46.317025 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:46.317028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.317967 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.317969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:46.318045 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:46.318049 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:20:46.318100 (+ 51us) client_negotiation.cc:240] Negotiation successful
0621 05:20:46.318135 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260621 05:20:46.318475 6738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.313840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40538 (local address 127.2.32.2:37829)
0621 05:20:46.314853 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:46.314857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:46.314864 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:20:46.314896 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:46.314898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:46.314909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:46.314993 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:46.315114 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.315904 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.317113 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.317818 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.317888 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:46.317936 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:46.318342 (+ 406us) server_negotiation.cc:299] Negotiation successful
0621 05:20:46.318380 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":985,"thread_start_us":84,"threads_started":1}
I20260621 05:20:46.318485 6656 tablet_service.cc:1467] Processing CreateTablet for tablet 607a0f14598a43bcbbd25692742a2c50 (DEFAULT_TABLE table=test-table [id=da2c403bf1c94f3aa87491c897b1f070]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:46.318815 6656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 607a0f14598a43bcbbd25692742a2c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:46.319043 6523 tablet_service.cc:1467] Processing CreateTablet for tablet 607a0f14598a43bcbbd25692742a2c50 (DEFAULT_TABLE table=test-table [id=da2c403bf1c94f3aa87491c897b1f070]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:46.319494 6523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 607a0f14598a43bcbbd25692742a2c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:46.321516 6739 tablet_bootstrap.cc:492] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31: Bootstrap starting.
I20260621 05:20:46.321909 6390 tablet_service.cc:1467] Processing CreateTablet for tablet 607a0f14598a43bcbbd25692742a2c50 (DEFAULT_TABLE table=test-table [id=da2c403bf1c94f3aa87491c897b1f070]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:46.322201 6739 tablet_bootstrap.cc:654] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:46.322238 6390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 607a0f14598a43bcbbd25692742a2c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:46.322808 6739 log.cc:824] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:46.324163 6741 tablet_bootstrap.cc:492] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d: Bootstrap starting.
I20260621 05:20:46.324776 6739 tablet_bootstrap.cc:492] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31: No bootstrap required, opened a new log
I20260621 05:20:46.324805 6741 tablet_bootstrap.cc:654] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:46.324857 6739 ts_tablet_manager.cc:1397] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:20:46.325340 6742 tablet_bootstrap.cc:492] T 607a0f14598a43bcbbd25692742a2c50 P 3cf785c338ba4e38abdf761425728097: Bootstrap starting.
I20260621 05:20:46.325362 6741 log.cc:824] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:46.326042 6742 tablet_bootstrap.cc:654] T 607a0f14598a43bcbbd25692742a2c50 P 3cf785c338ba4e38abdf761425728097: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:46.326196 6741 tablet_bootstrap.cc:492] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d: No bootstrap required, opened a new log
I20260621 05:20:46.326282 6741 ts_tablet_manager.cc:1397] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:20:46.326736 6742 log.cc:824] T 607a0f14598a43bcbbd25692742a2c50 P 3cf785c338ba4e38abdf761425728097: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:46.327217 6739 raft_consensus.cc:348] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf3325a2ff2d4659a0deb2f4cbab0e31" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41253 } } peers { permanent_uuid: "c7cea9461e3644bc9bec463c389a561d" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37829 } } peers { permanent_uuid: "3cf785c338ba4e38abdf761425728097" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 39271 } }
I20260621 05:20:46.327400 6739 raft_consensus.cc:374] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:46.327437 6739 raft_consensus.cc:729] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf3325a2ff2d4659a0deb2f4cbab0e31, State: Initialized, Role: FOLLOWER
I20260621 05:20:46.327549 6742 tablet_bootstrap.cc:492] T 607a0f14598a43bcbbd25692742a2c50 P 3cf785c338ba4e38abdf761425728097: No bootstrap required, opened a new log
I20260621 05:20:46.327611 6742 ts_tablet_manager.cc:1397] T 607a0f14598a43bcbbd25692742a2c50 P 3cf785c338ba4e38abdf761425728097: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:20:46.327584 6739 consensus_queue.cc:260] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf3325a2ff2d4659a0deb2f4cbab0e31" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41253 } } peers { permanent_uuid: "c7cea9461e3644bc9bec463c389a561d" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37829 } } peers { permanent_uuid: "3cf785c338ba4e38abdf761425728097" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 39271 } }
I20260621 05:20:46.327854 6739 ts_tablet_manager.cc:1428] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:20:46.327867 6722 heartbeater.cc:503] Master 127.2.32.62:36493 was elected leader, sending a full tablet report...
I20260621 05:20:46.328322 6741 raft_consensus.cc:348] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf3325a2ff2d4659a0deb2f4cbab0e31" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41253 } } peers { permanent_uuid: "c7cea9461e3644bc9bec463c389a561d" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37829 } } peers { permanent_uuid: "3cf785c338ba4e38abdf761425728097" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 39271 } }
I20260621 05:20:46.328475 6741 raft_consensus.cc:374] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:46.328509 6741 raft_consensus.cc:729] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7cea9461e3644bc9bec463c389a561d, State: Initialized, Role: FOLLOWER
I20260621 05:20:46.328612 6741 consensus_queue.cc:260] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf3325a2ff2d4659a0deb2f4cbab0e31" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41253 } } peers { permanent_uuid: "c7cea9461e3644bc9bec463c389a561d" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37829 } } peers { permanent_uuid: "3cf785c338ba4e38abdf761425728097" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 39271 } }
I20260621 05:20:46.328877 6589 heartbeater.cc:503] Master 127.2.32.62:36493 was elected leader, sending a full tablet report...
I20260621 05:20:46.328877 6741 ts_tablet_manager.cc:1428] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:20:46.329612 6742 raft_consensus.cc:348] T 607a0f14598a43bcbbd25692742a2c50 P 3cf785c338ba4e38abdf761425728097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf3325a2ff2d4659a0deb2f4cbab0e31" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41253 } } peers { permanent_uuid: "c7cea9461e3644bc9bec463c389a561d" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37829 } } peers { permanent_uuid: "3cf785c338ba4e38abdf761425728097" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 39271 } }
I20260621 05:20:46.329803 6742 raft_consensus.cc:374] T 607a0f14598a43bcbbd25692742a2c50 P 3cf785c338ba4e38abdf761425728097 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:46.329854 6742 raft_consensus.cc:729] T 607a0f14598a43bcbbd25692742a2c50 P 3cf785c338ba4e38abdf761425728097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf785c338ba4e38abdf761425728097, State: Initialized, Role: FOLLOWER
I20260621 05:20:46.329967 6742 consensus_queue.cc:260] T 607a0f14598a43bcbbd25692742a2c50 P 3cf785c338ba4e38abdf761425728097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf3325a2ff2d4659a0deb2f4cbab0e31" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41253 } } peers { permanent_uuid: "c7cea9461e3644bc9bec463c389a561d" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37829 } } peers { permanent_uuid: "3cf785c338ba4e38abdf761425728097" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 39271 } }
I20260621 05:20:46.330194 6742 ts_tablet_manager.cc:1428] T 607a0f14598a43bcbbd25692742a2c50 P 3cf785c338ba4e38abdf761425728097: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:20:46.330260 6456 heartbeater.cc:503] Master 127.2.32.62:36493 was elected leader, sending a full tablet report...
W20260621 05:20:46.350577 6590 tablet.cc:2369] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:20:46.369184 6746 raft_consensus.cc:482] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:20:46.369400 6746 raft_consensus.cc:504] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf3325a2ff2d4659a0deb2f4cbab0e31" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41253 } } peers { permanent_uuid: "c7cea9461e3644bc9bec463c389a561d" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37829 } } peers { permanent_uuid: "3cf785c338ba4e38abdf761425728097" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 39271 } }
I20260621 05:20:46.370090 6746 leader_election.cc:288] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf3325a2ff2d4659a0deb2f4cbab0e31 (127.2.32.3:41253), 3cf785c338ba4e38abdf761425728097 (127.2.32.1:39271)
I20260621 05:20:46.374222 6735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.370450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:46003 (local address 127.2.32.3:41253)
0621 05:20:46.370489 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:46.370491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:46.370790 (+ 299us) server_negotiation.cc:407] Connection header received
0621 05:20:46.370830 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:46.370831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:46.370857 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:46.370971 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:46.371275 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.371834 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.373197 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.373764 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.373827 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:46.373837 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:46.374113 (+ 276us) server_negotiation.cc:299] Negotiation successful
0621 05:20:46.374150 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:20:46.374280 6591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.370316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:39271 (local address 127.2.32.2:35943)
0621 05:20:46.370444 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:46.370462 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:46.370541 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:46.370822 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:46.370825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:46.370828 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:46.370954 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:46.370957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.371847 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.371849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:46.373202 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:46.373207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.373916 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.373917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:46.373994 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:46.373998 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:20:46.374073 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 05:20:46.374138 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260621 05:20:46.374316 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.370356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:35943 (local address 127.2.32.1:39271)
0621 05:20:46.370411 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:46.370414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:46.370538 (+ 124us) server_negotiation.cc:407] Connection header received
0621 05:20:46.370635 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:46.370637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:46.370650 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:46.370733 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:46.371085 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.371726 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.373364 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.373793 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.373908 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:46.373919 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:46.374202 (+ 283us) server_negotiation.cc:299] Negotiation successful
0621 05:20:46.374249 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260621 05:20:46.374291 6748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.370329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:41253 (local address 127.2.32.2:46003)
0621 05:20:46.370663 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:46.370678 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:46.370758 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:46.371065 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:46.371066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:46.371069 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:46.371190 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:46.371192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.371962 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.371963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:46.373090 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:46.373093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.373867 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.373868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:46.373978 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:46.373984 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:20:46.374107 (+ 123us) client_negotiation.cc:240] Negotiation successful
0621 05:20:46.374163 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":14,"thread_start_us":170,"threads_started":1}
I20260621 05:20:46.374682 6410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607a0f14598a43bcbbd25692742a2c50" candidate_uuid: "c7cea9461e3644bc9bec463c389a561d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cf785c338ba4e38abdf761425728097" is_pre_election: true
I20260621 05:20:46.374699 6676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607a0f14598a43bcbbd25692742a2c50" candidate_uuid: "c7cea9461e3644bc9bec463c389a561d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf3325a2ff2d4659a0deb2f4cbab0e31" is_pre_election: true
I20260621 05:20:46.374854 6676 raft_consensus.cc:2454] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7cea9461e3644bc9bec463c389a561d in term 0.
I20260621 05:20:46.374854 6410 raft_consensus.cc:2454] T 607a0f14598a43bcbbd25692742a2c50 P 3cf785c338ba4e38abdf761425728097 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7cea9461e3644bc9bec463c389a561d in term 0.
I20260621 05:20:46.375118 6476 leader_election.cc:302] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cf785c338ba4e38abdf761425728097, c7cea9461e3644bc9bec463c389a561d; no voters:
I20260621 05:20:46.375352 6746 raft_consensus.cc:2781] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:20:46.375418 6746 raft_consensus.cc:482] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:20:46.375437 6746 raft_consensus.cc:3037] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:46.376101 6746 raft_consensus.cc:504] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf3325a2ff2d4659a0deb2f4cbab0e31" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41253 } } peers { permanent_uuid: "c7cea9461e3644bc9bec463c389a561d" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37829 } } peers { permanent_uuid: "3cf785c338ba4e38abdf761425728097" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 39271 } }
I20260621 05:20:46.376261 6746 leader_election.cc:288] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [CANDIDATE]: Term 1 election: Requested vote from peers cf3325a2ff2d4659a0deb2f4cbab0e31 (127.2.32.3:41253), 3cf785c338ba4e38abdf761425728097 (127.2.32.1:39271)
I20260621 05:20:46.376394 6676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607a0f14598a43bcbbd25692742a2c50" candidate_uuid: "c7cea9461e3644bc9bec463c389a561d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf3325a2ff2d4659a0deb2f4cbab0e31"
I20260621 05:20:46.376480 6676 raft_consensus.cc:3037] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:46.376477 6410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607a0f14598a43bcbbd25692742a2c50" candidate_uuid: "c7cea9461e3644bc9bec463c389a561d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cf785c338ba4e38abdf761425728097"
I20260621 05:20:46.376569 6410 raft_consensus.cc:3037] T 607a0f14598a43bcbbd25692742a2c50 P 3cf785c338ba4e38abdf761425728097 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:46.377180 6676 raft_consensus.cc:2454] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7cea9461e3644bc9bec463c389a561d in term 1.
I20260621 05:20:46.377194 6410 raft_consensus.cc:2454] T 607a0f14598a43bcbbd25692742a2c50 P 3cf785c338ba4e38abdf761425728097 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7cea9461e3644bc9bec463c389a561d in term 1.
I20260621 05:20:46.377380 6477 leader_election.cc:302] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7cea9461e3644bc9bec463c389a561d, cf3325a2ff2d4659a0deb2f4cbab0e31; no voters:
I20260621 05:20:46.377470 6746 raft_consensus.cc:2781] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:46.377622 6746 raft_consensus.cc:686] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [term 1 LEADER]: Becoming Leader. State: Replica: c7cea9461e3644bc9bec463c389a561d, State: Running, Role: LEADER
I20260621 05:20:46.377717 6746 consensus_queue.cc:237] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf3325a2ff2d4659a0deb2f4cbab0e31" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41253 } } peers { permanent_uuid: "c7cea9461e3644bc9bec463c389a561d" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37829 } } peers { permanent_uuid: "3cf785c338ba4e38abdf761425728097" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 39271 } }
I20260621 05:20:46.378536 6265 catalog_manager.cc:5495] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d reported cstate change: term changed from 0 to 1, leader changed from <none> to c7cea9461e3644bc9bec463c389a561d (127.2.32.2). New cstate: current_term: 1 leader_uuid: "c7cea9461e3644bc9bec463c389a561d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf3325a2ff2d4659a0deb2f4cbab0e31" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7cea9461e3644bc9bec463c389a561d" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cf785c338ba4e38abdf761425728097" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 39271 } health_report { overall_health: UNKNOWN } } }
I20260621 05:20:46.393311 6738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.390410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.2.32.2:37829)
0621 05:20:46.390467 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:46.390470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:46.390479 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:20:46.390654 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:46.390655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:46.390678 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:46.390753 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:46.391010 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.391599 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.392472 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:46.392677 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:46.392740 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:46.392919 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:46.393012 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:46.393076 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:46.393164 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 05:20:46.393184 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260621 05:20:46.395532 6410 raft_consensus.cc:1261] T 607a0f14598a43bcbbd25692742a2c50 P 3cf785c338ba4e38abdf761425728097 [term 1 FOLLOWER]: Refusing update from remote peer c7cea9461e3644bc9bec463c389a561d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:46.395614 6676 raft_consensus.cc:1261] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31 [term 1 FOLLOWER]: Refusing update from remote peer c7cea9461e3644bc9bec463c389a561d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:46.395928 6746 consensus_queue.cc:1035] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cf785c338ba4e38abdf761425728097" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:46.396090 6749 consensus_queue.cc:1035] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf3325a2ff2d4659a0deb2f4cbab0e31" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:46.398675 6753 mvcc.cc:205] Tried to move back new op lower bound from 7299150833232728064 to 7299150833163538432. Current Snapshot: MvccSnapshot[applied={T|T < 7299150833232728064}]
I20260621 05:20:46.398818 6752 mvcc.cc:205] Tried to move back new op lower bound from 7299150833232728064 to 7299150833163538432. Current Snapshot: MvccSnapshot[applied={T|T < 7299150833232728064}]
I20260621 05:20:46.400748 6265 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:20:46.400827 6265 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:20:46.401593 6265 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 323e46ba44ea45a8839f1a3164619a70: Sending DeleteTablet for 3 replicas of tablet 607a0f14598a43bcbbd25692742a2c50
I20260621 05:20:46.402251 6656 tablet_service.cc:1514] Processing DeleteTablet for tablet 607a0f14598a43bcbbd25692742a2c50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:46 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:58574
I20260621 05:20:46.402354 6523 tablet_service.cc:1514] Processing DeleteTablet for tablet 607a0f14598a43bcbbd25692742a2c50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:46 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:40538
I20260621 05:20:46.402714 6763 tablet_replica.cc:330] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d: stopping tablet replica
I20260621 05:20:46.402843 6763 raft_consensus.cc:2229] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:20:46.403014 6763 raft_consensus.cc:2258] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:46.403012 6390 tablet_service.cc:1514] Processing DeleteTablet for tablet 607a0f14598a43bcbbd25692742a2c50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:46 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:37544
I20260621 05:20:46.403393 6764 tablet_replica.cc:330] T 607a0f14598a43bcbbd25692742a2c50 P 3cf785c338ba4e38abdf761425728097: stopping tablet replica
I20260621 05:20:46.403497 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 6328
I20260621 05:20:46.403589 6763 ts_tablet_manager.cc:1905] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:46.405248 6762 tablet_replica.cc:330] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31: stopping tablet replica
I20260621 05:20:46.405315 6763 ts_tablet_manager.cc:1918] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:20:46.405390 6763 log.cc:1196] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/607a0f14598a43bcbbd25692742a2c50
I20260621 05:20:46.405524 6762 raft_consensus.cc:2229] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:20:46.405658 6763 ts_tablet_manager.cc:1939] T 607a0f14598a43bcbbd25692742a2c50 P c7cea9461e3644bc9bec463c389a561d: Deleting consensus metadata
I20260621 05:20:46.405690 6762 raft_consensus.cc:2258] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:46.406212 6251 catalog_manager.cc:4841] TS c7cea9461e3644bc9bec463c389a561d (127.2.32.2:37829): tablet 607a0f14598a43bcbbd25692742a2c50 (table test-table [id=da2c403bf1c94f3aa87491c897b1f070]) successfully deleted
I20260621 05:20:46.406438 6762 ts_tablet_manager.cc:1905] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:46.408326 6762 ts_tablet_manager.cc:1918] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:20:46.408399 6762 log.cc:1196] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/607a0f14598a43bcbbd25692742a2c50
I20260621 05:20:46.408644 6762 ts_tablet_manager.cc:1939] T 607a0f14598a43bcbbd25692742a2c50 P cf3325a2ff2d4659a0deb2f4cbab0e31: Deleting consensus metadata
I20260621 05:20:46.409037 6251 catalog_manager.cc:4841] TS cf3325a2ff2d4659a0deb2f4cbab0e31 (127.2.32.3:41253): tablet 607a0f14598a43bcbbd25692742a2c50 (table test-table [id=da2c403bf1c94f3aa87491c897b1f070]) successfully deleted
W20260621 05:20:46.409993 6250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.32.1:39271 (error 108)
W20260621 05:20:46.410470 6736 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:46.410329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:39271 (local address 127.0.0.1:37546)
0621 05:20:46.410389 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:46.410431 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.1:39271: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":39}
W20260621 05:20:46.410575 6250 catalog_manager.cc:4584] TS 3cf785c338ba4e38abdf761425728097 (127.2.32.1:39271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 607a0f14598a43bcbbd25692742a2c50: Network error: Client connection negotiation failed: client connection to 127.2.32.1:39271: connect: Connection reset by peer (error 104)
I20260621 05:20:46.410794 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 6461
I20260621 05:20:46.415580 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 6594
I20260621 05:20:46.420655 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 6235
2026-06-21T05:20:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2042 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-21T05:20:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:46Z Disabled control of system clock
I20260621 05:20:46.448854 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:44207
--webserver_interface=127.2.32.62
--webserver_port=0
--builtin_ntp_servers=127.2.32.20:40507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:44207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 05:20:46.533159 6769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:46.533413 6769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:46.533440 6769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:46.534507 6769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:46.534547 6769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:46.534560 6769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:46.534574 6769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:46.534585 6769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:46.535910 6769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:40507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:44207
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:44207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:46.536104 6769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:46.536325 6769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:46.541033 6775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:46.541136 6777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:46.541146 6769 server_base.cc:1034] running on GCE node
W20260621 05:20:46.541059 6774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:46.541572 6769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:46.541843 6769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:46.543018 6769 hybrid_clock.cc:648] HybridClock initialized: now 1782019246542999 us; error 30 us; skew 500 ppm
I20260621 05:20:46.544749 6769 webserver.cc:460] Webserver started at http://127.2.32.62:46695/ using document root <none> and password file <none>
I20260621 05:20:46.545099 6769 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:46.545176 6769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:46.545374 6769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:46.546646 6769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "8c859faa41b04f03beebb21edea76d9d"
format_stamp: "Formatted at 2026-06-21 05:20:46 on dist-test-slave-b357"
server_key: "537a93444e15b8f05816b4092f20864c"
server_key_iv: "208dfa4724ecd2d0f1d1bb68f692bcc3"
server_key_version: "encryptionkey@0"
I20260621 05:20:46.547051 6769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "8c859faa41b04f03beebb21edea76d9d"
format_stamp: "Formatted at 2026-06-21 05:20:46 on dist-test-slave-b357"
server_key: "537a93444e15b8f05816b4092f20864c"
server_key_iv: "208dfa4724ecd2d0f1d1bb68f692bcc3"
server_key_version: "encryptionkey@0"
I20260621 05:20:46.548781 6769 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:46.549763 6783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:46.549942 6769 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:46.550050 6769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "8c859faa41b04f03beebb21edea76d9d"
format_stamp: "Formatted at 2026-06-21 05:20:46 on dist-test-slave-b357"
server_key: "537a93444e15b8f05816b4092f20864c"
server_key_iv: "208dfa4724ecd2d0f1d1bb68f692bcc3"
server_key_version: "encryptionkey@0"
I20260621 05:20:46.550149 6769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:46.554368 6769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:46.554626 6769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:46.554774 6769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:46.558542 6769 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:44207
I20260621 05:20:46.558656 6835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:44207 every 8 connection(s)
I20260621 05:20:46.558924 6769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:46.559607 6836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:46.562237 6836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d: Bootstrap starting.
I20260621 05:20:46.562795 6836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:46.563118 6836 log.cc:824] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:46.564040 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 6769
I20260621 05:20:46.564177 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:20:46.564330 6836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d: No bootstrap required, opened a new log
I20260621 05:20:46.564424 2176 external_mini_cluster.cc:1428] Setting key 7950b96e643f92da723c9e23050aac66
I20260621 05:20:46.567209 6836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c859faa41b04f03beebb21edea76d9d" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 44207 } }
I20260621 05:20:46.567375 6836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:46.567421 6836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c859faa41b04f03beebb21edea76d9d, State: Initialized, Role: FOLLOWER
I20260621 05:20:46.567567 6836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c859faa41b04f03beebb21edea76d9d" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 44207 } }
I20260621 05:20:46.567669 6836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:46.567759 6836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:46.567811 6836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:46.567819 6840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.565165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35530 (local address 127.2.32.62:44207)
0621 05:20:46.565444 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:46.565454 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:46.565488 (+ 34us) server_negotiation.cc:407] Connection header received
0621 05:20:46.565999 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:46.566007 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:46.566052 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:46.566271 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:46.567211 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:46.567220 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:46.567220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:46.567225 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:46.567280 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:46.567292 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:46.567374 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:46.567451 (+ 77us) server_negotiation.cc:299] Negotiation successful
0621 05:20:46.567611 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":118,"threads_started":1}
I20260621 05:20:46.568393 6836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c859faa41b04f03beebb21edea76d9d" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 44207 } }
I20260621 05:20:46.568506 6836 leader_election.cc:302] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c859faa41b04f03beebb21edea76d9d; no voters:
I20260621 05:20:46.568663 6836 leader_election.cc:288] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:46.568696 6841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:46.568853 6841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d [term 1 LEADER]: Becoming Leader. State: Replica: 8c859faa41b04f03beebb21edea76d9d, State: Running, Role: LEADER
I20260621 05:20:46.568929 6836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:46.568929 6841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c859faa41b04f03beebb21edea76d9d" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 44207 } }
I20260621 05:20:46.569423 6843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c859faa41b04f03beebb21edea76d9d. Latest consensus state: current_term: 1 leader_uuid: "8c859faa41b04f03beebb21edea76d9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c859faa41b04f03beebb21edea76d9d" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 44207 } } }
I20260621 05:20:46.569433 6842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c859faa41b04f03beebb21edea76d9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c859faa41b04f03beebb21edea76d9d" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 44207 } } }
I20260621 05:20:46.569587 6843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:46.569597 6842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d [sys.catalog]: This master's current role is: LEADER
W20260621 05:20:46.570986 6857 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:20:46.571053 6857 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:20:46.571168 6850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:46.571522 6850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:46.573725 6850 catalog_manager.cc:1347] Generated new cluster ID: 639ac902ec3d44b988261a976c52b005
I20260621 05:20:46.573802 6850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:46.583525 6850 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:46.583689 6850 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:46.588459 6850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d: Generated new TSK 0
I20260621 05:20:46.588635 6850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:20:46.595510 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:44207
--builtin_ntp_servers=127.2.32.20:40507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 05:20:46.691283 6860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:46.691510 6860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:46.691550 6860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:46.692711 6860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:46.692776 6860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:46.692823 6860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:20:46.694125 6860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:40507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:44207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:46.694394 6860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:46.694680 6860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:46.699595 6865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:46.699689 6866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:46.699595 6868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:46.699676 6860 server_base.cc:1034] running on GCE node
I20260621 05:20:46.699980 6860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:46.700232 6860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:46.701433 6860 hybrid_clock.cc:648] HybridClock initialized: now 1782019246701420 us; error 49 us; skew 500 ppm
I20260621 05:20:46.702785 6860 webserver.cc:460] Webserver started at http://127.2.32.1:37059/ using document root <none> and password file <none>
I20260621 05:20:46.703060 6860 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:46.703140 6860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:46.703336 6860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:46.704468 6860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "87f9ca0f5f234bac8f06c90a74166705"
format_stamp: "Formatted at 2026-06-21 05:20:46 on dist-test-slave-b357"
server_key: "79710f6a119089e76a99a450fb86d840"
server_key_iv: "9a3863df56885b4a4be38e7c357295a0"
server_key_version: "encryptionkey@0"
I20260621 05:20:46.704879 6860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "87f9ca0f5f234bac8f06c90a74166705"
format_stamp: "Formatted at 2026-06-21 05:20:46 on dist-test-slave-b357"
server_key: "79710f6a119089e76a99a450fb86d840"
server_key_iv: "9a3863df56885b4a4be38e7c357295a0"
server_key_version: "encryptionkey@0"
I20260621 05:20:46.706355 6860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:20:46.707288 6874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:46.707547 6860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:46.707648 6860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "87f9ca0f5f234bac8f06c90a74166705"
format_stamp: "Formatted at 2026-06-21 05:20:46 on dist-test-slave-b357"
server_key: "79710f6a119089e76a99a450fb86d840"
server_key_iv: "9a3863df56885b4a4be38e7c357295a0"
server_key_version: "encryptionkey@0"
I20260621 05:20:46.707731 6860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:46.711028 6860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:46.711339 6860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:46.711485 6860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:46.711696 6860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:46.711985 6860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:46.712038 6860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:46.712076 6860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:46.712103 6860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:46.718683 6860 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:40843
I20260621 05:20:46.718734 6987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:40843 every 8 connection(s)
I20260621 05:20:46.719024 6860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:20:46.721597 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 6860
I20260621 05:20:46.721747 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:20:46.721975 2176 external_mini_cluster.cc:1428] Setting key 535b25403bbaa3cd40b38e7ad1acf26a
I20260621 05:20:46.722642 6840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.719808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:49093 (local address 127.2.32.62:44207)
0621 05:20:46.719924 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:46.719927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:46.720013 (+ 86us) server_negotiation.cc:407] Connection header received
0621 05:20:46.720475 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:46.720477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:46.720485 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:46.720564 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:46.722300 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:46.722302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:46.722303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:46.722311 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:46.722330 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:46.722335 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:46.722403 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:46.722562 (+ 159us) server_negotiation.cc:299] Negotiation successful
0621 05:20:46.722578 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 05:20:46.722786 6990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.719712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:44207 (local address 127.2.32.1:49093)
0621 05:20:46.719887 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:46.719925 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:46.720345 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:46.720829 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:46.720832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:46.720840 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:46.721998 (+ 1158us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:20:46.721999 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:46.722050 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:20:46.722086 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:46.722436 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:46.722443 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:46.722446 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:20:46.722540 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 05:20:46.722589 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260621 05:20:46.723304 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:44207
--builtin_ntp_servers=127.2.32.20:40507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260621 05:20:46.723424 6988 heartbeater.cc:347] Connected to a master server at 127.2.32.62:44207
I20260621 05:20:46.723521 6988 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:46.723668 6988 heartbeater.cc:511] Master 127.2.32.62:44207 requested a full tablet report, sending...
I20260621 05:20:46.724157 6800 ts_manager.cc:194] Registered new tserver with Master: 87f9ca0f5f234bac8f06c90a74166705 (127.2.32.1:40843)
W20260621 05:20:46.809626 6991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:46.809881 6991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:46.809904 6991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:46.811107 6991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:46.811199 6991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:46.811241 6991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:20:46.812495 6991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:40507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:44207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:46.812669 6991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:46.812880 6991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:46.817735 6999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:46.817725 6997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:46.817740 6991 server_base.cc:1034] running on GCE node
W20260621 05:20:46.817724 6996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:46.818214 6991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:46.818464 6991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:46.819651 6991 hybrid_clock.cc:648] HybridClock initialized: now 1782019246819649 us; error 56 us; skew 500 ppm
I20260621 05:20:46.821094 6991 webserver.cc:460] Webserver started at http://127.2.32.2:42205/ using document root <none> and password file <none>
I20260621 05:20:46.821339 6991 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:46.821411 6991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:46.821561 6991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:46.822571 6991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "266629fc39a24138aec672164860b401"
format_stamp: "Formatted at 2026-06-21 05:20:46 on dist-test-slave-b357"
server_key: "0ef19cb6fb323827e06e6b2f65b3af07"
server_key_iv: "914110716986b6606c6dce1456d5ce9d"
server_key_version: "encryptionkey@0"
I20260621 05:20:46.822939 6991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "266629fc39a24138aec672164860b401"
format_stamp: "Formatted at 2026-06-21 05:20:46 on dist-test-slave-b357"
server_key: "0ef19cb6fb323827e06e6b2f65b3af07"
server_key_iv: "914110716986b6606c6dce1456d5ce9d"
server_key_version: "encryptionkey@0"
I20260621 05:20:46.824340 6991 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:46.825114 7005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:46.825469 6991 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:46.825599 6991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "266629fc39a24138aec672164860b401"
format_stamp: "Formatted at 2026-06-21 05:20:46 on dist-test-slave-b357"
server_key: "0ef19cb6fb323827e06e6b2f65b3af07"
server_key_iv: "914110716986b6606c6dce1456d5ce9d"
server_key_version: "encryptionkey@0"
I20260621 05:20:46.825692 6991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:46.829010 6991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:46.829253 6991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:46.829396 6991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:46.829600 6991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:46.829906 6991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:46.829960 6991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:46.829999 6991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:46.830025 6991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:46.836452 6991 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:42155
I20260621 05:20:46.836493 7118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:42155 every 8 connection(s)
I20260621 05:20:46.836783 6991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:20:46.838529 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 6991
I20260621 05:20:46.838642 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:20:46.838869 2176 external_mini_cluster.cc:1428] Setting key 24dbb69cd118120dca4441054f99852d
I20260621 05:20:46.839829 6840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.837418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:39133 (local address 127.2.32.62:44207)
0621 05:20:46.837479 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:46.837481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:46.837709 (+ 228us) server_negotiation.cc:407] Connection header received
0621 05:20:46.838189 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:46.838192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:46.838201 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:46.838300 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:46.839601 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:46.839602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:46.839602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:46.839607 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:46.839626 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:46.839631 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:46.839685 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:46.839769 (+ 84us) server_negotiation.cc:299] Negotiation successful
0621 05:20:46.839781 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:20:46.839875 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:44207
--builtin_ntp_servers=127.2.32.20:40507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260621 05:20:46.840054 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.837407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:44207 (local address 127.2.32.2:39133)
0621 05:20:46.837582 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:46.837618 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:46.838023 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:46.838376 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:46.838380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:46.838390 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:46.839459 (+ 1069us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:20:46.839460 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:46.839485 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:20:46.839511 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:46.839687 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:46.839691 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:46.839694 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:20:46.839776 (+ 82us) client_negotiation.cc:240] Negotiation successful
0621 05:20:46.839835 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":96,"threads_started":1}
I20260621 05:20:46.841604 7119 heartbeater.cc:347] Connected to a master server at 127.2.32.62:44207
I20260621 05:20:46.841717 7119 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:46.841869 7119 heartbeater.cc:511] Master 127.2.32.62:44207 requested a full tablet report, sending...
I20260621 05:20:46.842226 6800 ts_manager.cc:194] Registered new tserver with Master: 266629fc39a24138aec672164860b401 (127.2.32.2:42155)
W20260621 05:20:46.930068 7122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:46.930277 7122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:46.930305 7122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:46.931521 7122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:46.931558 7122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:46.931583 7122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:20:46.932904 7122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:40507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:44207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:46.933125 7122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:46.933441 7122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:46.938299 7128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:46.938380 7130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:46.938421 7127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:46.938503 7122 server_base.cc:1034] running on GCE node
I20260621 05:20:46.938823 7122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:46.939060 7122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:46.940227 7122 hybrid_clock.cc:648] HybridClock initialized: now 1782019246940222 us; error 51 us; skew 500 ppm
I20260621 05:20:46.941807 7122 webserver.cc:460] Webserver started at http://127.2.32.3:38033/ using document root <none> and password file <none>
I20260621 05:20:46.942108 7122 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:46.942166 7122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:46.942327 7122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:46.943768 7122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "9bbf44c1a24e44d2bbaa6a99fca82bd4"
format_stamp: "Formatted at 2026-06-21 05:20:46 on dist-test-slave-b357"
server_key: "84cf75712c742a7072543f1fb6ff3737"
server_key_iv: "8a60063e0ff58cf850c5bc9f1e1e3e12"
server_key_version: "encryptionkey@0"
I20260621 05:20:46.944252 7122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "9bbf44c1a24e44d2bbaa6a99fca82bd4"
format_stamp: "Formatted at 2026-06-21 05:20:46 on dist-test-slave-b357"
server_key: "84cf75712c742a7072543f1fb6ff3737"
server_key_iv: "8a60063e0ff58cf850c5bc9f1e1e3e12"
server_key_version: "encryptionkey@0"
I20260621 05:20:46.945923 7122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:20:46.946992 7136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:46.947400 7122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:46.947510 7122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "9bbf44c1a24e44d2bbaa6a99fca82bd4"
format_stamp: "Formatted at 2026-06-21 05:20:46 on dist-test-slave-b357"
server_key: "84cf75712c742a7072543f1fb6ff3737"
server_key_iv: "8a60063e0ff58cf850c5bc9f1e1e3e12"
server_key_version: "encryptionkey@0"
I20260621 05:20:46.947569 7122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:46.950898 7122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:46.951112 7122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:46.951284 7122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:46.951452 7122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:46.951766 7122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:46.951825 7122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:46.951869 7122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:46.951913 7122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:46.959311 7122 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:45107
I20260621 05:20:46.959386 7249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:45107 every 8 connection(s)
I20260621 05:20:46.959677 7122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:20:46.963325 6840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.960474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:42867 (local address 127.2.32.62:44207)
0621 05:20:46.960537 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:46.960539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:46.960798 (+ 259us) server_negotiation.cc:407] Connection header received
0621 05:20:46.961201 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:46.961203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:46.961208 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:46.961284 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:46.962936 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:46.962937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:46.962937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:46.962941 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:46.962953 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:46.962956 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:46.963015 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:46.963252 (+ 237us) server_negotiation.cc:299] Negotiation successful
0621 05:20:46.963264 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 05:20:46.963587 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.960466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:44207 (local address 127.2.32.3:42867)
0621 05:20:46.960651 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:46.960681 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:46.961099 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:46.961478 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:46.961484 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:46.961496 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:46.962756 (+ 1260us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:20:46.962757 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:46.962785 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:20:46.962813 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:46.963067 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:46.963073 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:46.963076 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:20:46.963257 (+ 181us) client_negotiation.cc:240] Negotiation successful
0621 05:20:46.963306 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
I20260621 05:20:46.964200 7250 heartbeater.cc:347] Connected to a master server at 127.2.32.62:44207
I20260621 05:20:46.964285 7250 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:46.964385 7250 heartbeater.cc:511] Master 127.2.32.62:44207 requested a full tablet report, sending...
I20260621 05:20:46.964717 6800 ts_manager.cc:194] Registered new tserver with Master: 9bbf44c1a24e44d2bbaa6a99fca82bd4 (127.2.32.3:45107)
I20260621 05:20:46.965044 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 7122
I20260621 05:20:46.965153 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:20:46.965341 2176 external_mini_cluster.cc:1428] Setting key aee55f5b065e005a587e15359cd51d1d
I20260621 05:20:46.966197 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:20:46.968760 6840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.967476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.2.32.62:44207)
0621 05:20:46.967546 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:46.967548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:46.967646 (+ 98us) server_negotiation.cc:407] Connection header received
0621 05:20:46.967715 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:46.967716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:46.967720 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:46.967764 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:46.968525 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:46.968527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:46.968527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:46.968530 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:46.968541 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:46.968545 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:46.968589 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:46.968667 (+ 78us) server_negotiation.cc:299] Negotiation successful
0621 05:20:46.968677 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 05:20:46.970541 6800 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:20:46.972198 6800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:20:46.977010 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.974995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:42155 (local address 127.0.0.1:49430)
0621 05:20:46.975423 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:46.975463 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:46.975520 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:46.975990 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:46.975991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:46.976000 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:46.976551 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:20:46.976552 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:46.976559 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:20:46.976571 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:46.976857 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:46.976863 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:46.976866 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:20:46.976912 (+ 46us) client_negotiation.cc:240] Negotiation successful
0621 05:20:46.976927 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":123,"threads_started":1}
I20260621 05:20:46.977042 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.975433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:45107 (local address 127.0.0.1:45280)
0621 05:20:46.975669 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:46.975680 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:46.975720 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:46.975950 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:46.975956 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:46.975963 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:46.976477 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:20:46.976477 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:46.976491 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:20:46.976517 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:46.976862 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:46.976863 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:46.976866 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:20:46.976952 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 05:20:46.976963 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":105,"threads_started":1}
I20260621 05:20:46.977277 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.975261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49430 (local address 127.2.32.2:42155)
0621 05:20:46.975830 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:46.975835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:46.975845 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:20:46.975872 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:46.975874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:46.975882 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:46.976185 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:46.976717 (+ 532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:46.976722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:46.976723 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:46.976727 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:46.976756 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:46.976776 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:46.977130 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:46.977156 (+ 26us) server_negotiation.cc:299] Negotiation successful
0621 05:20:46.977174 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":74,"threads_started":1}
I20260621 05:20:46.977450 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.975666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45280 (local address 127.2.32.3:45107)
0621 05:20:46.975823 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:46.975827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:46.975836 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:20:46.975858 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:46.975860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:46.975867 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:46.975935 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:46.976604 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:46.976611 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:46.976611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:46.976615 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:46.976652 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:46.976701 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:46.976820 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:46.977329 (+ 509us) server_negotiation.cc:299] Negotiation successful
0621 05:20:46.977346 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260621 05:20:46.977706 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.975042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:40843 (local address 127.0.0.1:41142)
0621 05:20:46.975423 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:46.975463 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:46.975520 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:46.976072 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:46.976073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:46.976076 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:46.976743 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:20:46.976743 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:46.976749 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:20:46.976759 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:46.976989 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:46.976990 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:46.976991 (+ 1us) client_negotiation.cc:769] Sending connection context
0621 05:20:46.977039 (+ 48us) client_negotiation.cc:240] Negotiation successful
0621 05:20:46.977053 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":136,"threads_started":1}
I20260621 05:20:46.978058 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:46.975218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41142 (local address 127.2.32.1:40843)
0621 05:20:46.975436 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:46.975442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:46.975584 (+ 142us) server_negotiation.cc:407] Connection header received
0621 05:20:46.975652 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:46.975655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:46.975665 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:46.975741 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:46.976841 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:46.976848 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:46.976848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:46.976852 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:46.976888 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:46.976908 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:46.976963 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:46.977936 (+ 973us) server_negotiation.cc:299] Negotiation successful
0621 05:20:46.977960 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":96,"threads_started":1}
I20260621 05:20:46.978523 7053 tablet_service.cc:1467] Processing CreateTablet for tablet c8567825087340c88cc1184ae9d766ee (DEFAULT_TABLE table=test-table [id=9bf1c1056435494bb0860905101b4c4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:46.978611 7184 tablet_service.cc:1467] Processing CreateTablet for tablet c8567825087340c88cc1184ae9d766ee (DEFAULT_TABLE table=test-table [id=9bf1c1056435494bb0860905101b4c4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:46.978611 6922 tablet_service.cc:1467] Processing CreateTablet for tablet c8567825087340c88cc1184ae9d766ee (DEFAULT_TABLE table=test-table [id=9bf1c1056435494bb0860905101b4c4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:46.978924 6922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8567825087340c88cc1184ae9d766ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:46.978924 7184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8567825087340c88cc1184ae9d766ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:46.979070 7053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8567825087340c88cc1184ae9d766ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:46.982254 7264 tablet_bootstrap.cc:492] T c8567825087340c88cc1184ae9d766ee P 87f9ca0f5f234bac8f06c90a74166705: Bootstrap starting.
I20260621 05:20:46.983203 7264 tablet_bootstrap.cc:654] T c8567825087340c88cc1184ae9d766ee P 87f9ca0f5f234bac8f06c90a74166705: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:46.983538 7264 log.cc:824] T c8567825087340c88cc1184ae9d766ee P 87f9ca0f5f234bac8f06c90a74166705: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:46.983870 7265 tablet_bootstrap.cc:492] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4: Bootstrap starting.
I20260621 05:20:46.984491 7264 tablet_bootstrap.cc:492] T c8567825087340c88cc1184ae9d766ee P 87f9ca0f5f234bac8f06c90a74166705: No bootstrap required, opened a new log
I20260621 05:20:46.984530 7265 tablet_bootstrap.cc:654] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:46.984582 7264 ts_tablet_manager.cc:1397] T c8567825087340c88cc1184ae9d766ee P 87f9ca0f5f234bac8f06c90a74166705: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:20:46.984839 7265 log.cc:824] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:46.985200 7266 tablet_bootstrap.cc:492] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401: Bootstrap starting.
I20260621 05:20:46.985903 7266 tablet_bootstrap.cc:654] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:46.986227 7266 log.cc:824] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:46.987049 7265 tablet_bootstrap.cc:492] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4: No bootstrap required, opened a new log
I20260621 05:20:46.987185 7265 ts_tablet_manager.cc:1397] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:20:46.987495 7266 tablet_bootstrap.cc:492] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401: No bootstrap required, opened a new log
I20260621 05:20:46.987581 7266 ts_tablet_manager.cc:1397] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:20:46.987536 7264 raft_consensus.cc:348] T c8567825087340c88cc1184ae9d766ee P 87f9ca0f5f234bac8f06c90a74166705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266629fc39a24138aec672164860b401" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42155 } } peers { permanent_uuid: "9bbf44c1a24e44d2bbaa6a99fca82bd4" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45107 } } peers { permanent_uuid: "87f9ca0f5f234bac8f06c90a74166705" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40843 } }
I20260621 05:20:46.987711 7264 raft_consensus.cc:374] T c8567825087340c88cc1184ae9d766ee P 87f9ca0f5f234bac8f06c90a74166705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:46.987756 7264 raft_consensus.cc:729] T c8567825087340c88cc1184ae9d766ee P 87f9ca0f5f234bac8f06c90a74166705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f9ca0f5f234bac8f06c90a74166705, State: Initialized, Role: FOLLOWER
I20260621 05:20:46.987880 7264 consensus_queue.cc:260] T c8567825087340c88cc1184ae9d766ee P 87f9ca0f5f234bac8f06c90a74166705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266629fc39a24138aec672164860b401" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42155 } } peers { permanent_uuid: "9bbf44c1a24e44d2bbaa6a99fca82bd4" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45107 } } peers { permanent_uuid: "87f9ca0f5f234bac8f06c90a74166705" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40843 } }
I20260621 05:20:46.988155 7264 ts_tablet_manager.cc:1428] T c8567825087340c88cc1184ae9d766ee P 87f9ca0f5f234bac8f06c90a74166705: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260621 05:20:46.988308 6988 heartbeater.cc:503] Master 127.2.32.62:44207 was elected leader, sending a full tablet report...
I20260621 05:20:46.989583 7265 raft_consensus.cc:348] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266629fc39a24138aec672164860b401" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42155 } } peers { permanent_uuid: "9bbf44c1a24e44d2bbaa6a99fca82bd4" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45107 } } peers { permanent_uuid: "87f9ca0f5f234bac8f06c90a74166705" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40843 } }
I20260621 05:20:46.989722 7265 raft_consensus.cc:374] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:46.989745 7265 raft_consensus.cc:729] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bbf44c1a24e44d2bbaa6a99fca82bd4, State: Initialized, Role: FOLLOWER
I20260621 05:20:46.989861 7265 consensus_queue.cc:260] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266629fc39a24138aec672164860b401" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42155 } } peers { permanent_uuid: "9bbf44c1a24e44d2bbaa6a99fca82bd4" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45107 } } peers { permanent_uuid: "87f9ca0f5f234bac8f06c90a74166705" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40843 } }
I20260621 05:20:46.990135 7265 ts_tablet_manager.cc:1428] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:20:46.990168 7250 heartbeater.cc:503] Master 127.2.32.62:44207 was elected leader, sending a full tablet report...
I20260621 05:20:46.990164 7266 raft_consensus.cc:348] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266629fc39a24138aec672164860b401" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42155 } } peers { permanent_uuid: "9bbf44c1a24e44d2bbaa6a99fca82bd4" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45107 } } peers { permanent_uuid: "87f9ca0f5f234bac8f06c90a74166705" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40843 } }
I20260621 05:20:46.990317 7266 raft_consensus.cc:374] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:46.990355 7266 raft_consensus.cc:729] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 266629fc39a24138aec672164860b401, State: Initialized, Role: FOLLOWER
I20260621 05:20:46.990451 7266 consensus_queue.cc:260] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266629fc39a24138aec672164860b401" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42155 } } peers { permanent_uuid: "9bbf44c1a24e44d2bbaa6a99fca82bd4" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45107 } } peers { permanent_uuid: "87f9ca0f5f234bac8f06c90a74166705" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40843 } }
I20260621 05:20:46.990711 7266 ts_tablet_manager.cc:1428] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:20:46.990834 7119 heartbeater.cc:503] Master 127.2.32.62:44207 was elected leader, sending a full tablet report...
I20260621 05:20:47.011925 7272 raft_consensus.cc:482] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:20:47.012074 7272 raft_consensus.cc:504] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266629fc39a24138aec672164860b401" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42155 } } peers { permanent_uuid: "9bbf44c1a24e44d2bbaa6a99fca82bd4" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45107 } } peers { permanent_uuid: "87f9ca0f5f234bac8f06c90a74166705" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40843 } }
I20260621 05:20:47.012442 7272 leader_election.cc:288] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bbf44c1a24e44d2bbaa6a99fca82bd4 (127.2.32.3:45107), 87f9ca0f5f234bac8f06c90a74166705 (127.2.32.1:40843)
I20260621 05:20:47.013962 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:47.012606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:40843 (local address 127.2.32.2:53009)
0621 05:20:47.012656 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:47.012668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:47.012706 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:47.012898 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:47.012899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:47.012901 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:47.013399 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:20:47.013400 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:47.013409 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:20:47.013426 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:47.013775 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:47.013776 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:47.013776 (+ 0us) client_negotiation.cc:769] Sending connection context
0621 05:20:47.013835 (+ 59us) client_negotiation.cc:240] Negotiation successful
0621 05:20:47.013876 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260621 05:20:47.014012 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:47.012668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:53009 (local address 127.2.32.1:40843)
0621 05:20:47.012750 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:47.012753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:47.012766 (+ 13us) server_negotiation.cc:407] Connection header received
0621 05:20:47.012803 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:47.012804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:47.012811 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:47.012890 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:47.013630 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:47.013631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:47.013632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:47.013638 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:47.013662 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:47.013668 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:47.013756 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:47.013838 (+ 82us) server_negotiation.cc:299] Negotiation successful
0621 05:20:47.013856 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:20:47.014484 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:47.012804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:57977 (local address 127.2.32.3:45107)
0621 05:20:47.012872 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:47.012874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:47.013140 (+ 266us) server_negotiation.cc:407] Connection header received
0621 05:20:47.013181 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:47.013181 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:47.013187 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:47.013252 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:47.014172 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:47.014173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:47.014174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:47.014178 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:47.014194 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:47.014199 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:47.014238 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:47.014389 (+ 151us) server_negotiation.cc:299] Negotiation successful
0621 05:20:47.014405 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 05:20:47.014418 6942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8567825087340c88cc1184ae9d766ee" candidate_uuid: "266629fc39a24138aec672164860b401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f9ca0f5f234bac8f06c90a74166705" is_pre_election: true
I20260621 05:20:47.014590 6942 raft_consensus.cc:2454] T c8567825087340c88cc1184ae9d766ee P 87f9ca0f5f234bac8f06c90a74166705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 266629fc39a24138aec672164860b401 in term 0.
I20260621 05:20:47.014691 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:47.012826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:45107 (local address 127.2.32.2:57977)
0621 05:20:47.013061 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:47.013072 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:47.013138 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:47.013375 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:47.013377 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:47.013382 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:47.014035 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:20:47.014035 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:47.014042 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:20:47.014055 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:47.014309 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:47.014310 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:47.014311 (+ 1us) client_negotiation.cc:769] Sending connection context
0621 05:20:47.014587 (+ 276us) client_negotiation.cc:240] Negotiation successful
0621 05:20:47.014603 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":171,"threads_started":1}
I20260621 05:20:47.014968 7007 leader_election.cc:302] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 266629fc39a24138aec672164860b401, 87f9ca0f5f234bac8f06c90a74166705; no voters:
I20260621 05:20:47.015180 7272 raft_consensus.cc:2781] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:20:47.015235 7272 raft_consensus.cc:482] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:20:47.015264 7272 raft_consensus.cc:3037] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:47.015323 7204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8567825087340c88cc1184ae9d766ee" candidate_uuid: "266629fc39a24138aec672164860b401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bbf44c1a24e44d2bbaa6a99fca82bd4" is_pre_election: true
I20260621 05:20:47.015472 7204 raft_consensus.cc:2454] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 266629fc39a24138aec672164860b401 in term 0.
I20260621 05:20:47.016170 7272 raft_consensus.cc:504] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266629fc39a24138aec672164860b401" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42155 } } peers { permanent_uuid: "9bbf44c1a24e44d2bbaa6a99fca82bd4" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45107 } } peers { permanent_uuid: "87f9ca0f5f234bac8f06c90a74166705" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40843 } }
I20260621 05:20:47.016302 7272 leader_election.cc:288] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [CANDIDATE]: Term 1 election: Requested vote from peers 9bbf44c1a24e44d2bbaa6a99fca82bd4 (127.2.32.3:45107), 87f9ca0f5f234bac8f06c90a74166705 (127.2.32.1:40843)
I20260621 05:20:47.016431 7204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8567825087340c88cc1184ae9d766ee" candidate_uuid: "266629fc39a24138aec672164860b401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bbf44c1a24e44d2bbaa6a99fca82bd4"
I20260621 05:20:47.016466 6942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8567825087340c88cc1184ae9d766ee" candidate_uuid: "266629fc39a24138aec672164860b401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f9ca0f5f234bac8f06c90a74166705"
I20260621 05:20:47.016505 7204 raft_consensus.cc:3037] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:47.016566 6942 raft_consensus.cc:3037] T c8567825087340c88cc1184ae9d766ee P 87f9ca0f5f234bac8f06c90a74166705 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:47.017184 7204 raft_consensus.cc:2454] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 266629fc39a24138aec672164860b401 in term 1.
I20260621 05:20:47.017336 6942 raft_consensus.cc:2454] T c8567825087340c88cc1184ae9d766ee P 87f9ca0f5f234bac8f06c90a74166705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 266629fc39a24138aec672164860b401 in term 1.
I20260621 05:20:47.017354 7009 leader_election.cc:302] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 266629fc39a24138aec672164860b401, 9bbf44c1a24e44d2bbaa6a99fca82bd4; no voters:
I20260621 05:20:47.017464 7272 raft_consensus.cc:2781] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:47.017619 7272 raft_consensus.cc:686] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [term 1 LEADER]: Becoming Leader. State: Replica: 266629fc39a24138aec672164860b401, State: Running, Role: LEADER
I20260621 05:20:47.017733 7272 consensus_queue.cc:237] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266629fc39a24138aec672164860b401" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42155 } } peers { permanent_uuid: "9bbf44c1a24e44d2bbaa6a99fca82bd4" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45107 } } peers { permanent_uuid: "87f9ca0f5f234bac8f06c90a74166705" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40843 } }
I20260621 05:20:47.019244 6800 catalog_manager.cc:5495] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 266629fc39a24138aec672164860b401 (127.2.32.2). New cstate: current_term: 1 leader_uuid: "266629fc39a24138aec672164860b401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266629fc39a24138aec672164860b401" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bbf44c1a24e44d2bbaa6a99fca82bd4" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87f9ca0f5f234bac8f06c90a74166705" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40843 } health_report { overall_health: UNKNOWN } } }
I20260621 05:20:47.024859 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:47.023552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49436 (local address 127.2.32.2:42155)
0621 05:20:47.023645 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:47.023647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:47.023658 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:20:47.023738 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:47.023739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:47.023747 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:47.023793 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:47.024608 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:47.024609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:47.024609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:47.024613 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:47.024626 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:47.024629 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:47.024693 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:47.024743 (+ 50us) server_negotiation.cc:299] Negotiation successful
0621 05:20:47.024754 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 05:20:47.026785 7204 raft_consensus.cc:1261] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4 [term 1 FOLLOWER]: Refusing update from remote peer 266629fc39a24138aec672164860b401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:47.027016 6942 raft_consensus.cc:1261] T c8567825087340c88cc1184ae9d766ee P 87f9ca0f5f234bac8f06c90a74166705 [term 1 FOLLOWER]: Refusing update from remote peer 266629fc39a24138aec672164860b401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:47.027092 7272 consensus_queue.cc:1035] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bbf44c1a24e44d2bbaa6a99fca82bd4" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 45107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:47.027390 7272 consensus_queue.cc:1035] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87f9ca0f5f234bac8f06c90a74166705" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:47.030495 7279 mvcc.cc:205] Tried to move back new op lower bound from 7299150835818901504 to 7299150835786620928. Current Snapshot: MvccSnapshot[applied={T|T < 7299150835818901504}]
I20260621 05:20:47.031845 6799 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:20:47.031937 6799 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:20:47.032675 6799 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c859faa41b04f03beebb21edea76d9d: Sending DeleteTablet for 3 replicas of tablet c8567825087340c88cc1184ae9d766ee
I20260621 05:20:47.032991 7053 tablet_service.cc:1514] Processing DeleteTablet for tablet c8567825087340c88cc1184ae9d766ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:47 UTC) from {username='slave'} at 127.0.0.1:49430
I20260621 05:20:47.033051 7184 tablet_service.cc:1514] Processing DeleteTablet for tablet c8567825087340c88cc1184ae9d766ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:47 UTC) from {username='slave'} at 127.0.0.1:45280
I20260621 05:20:47.033296 6922 tablet_service.cc:1514] Processing DeleteTablet for tablet c8567825087340c88cc1184ae9d766ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:47 UTC) from {username='slave'} at 127.0.0.1:41142
I20260621 05:20:47.033406 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 6860
I20260621 05:20:47.035681 7289 tablet_replica.cc:330] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4: stopping tablet replica
I20260621 05:20:47.035795 7289 raft_consensus.cc:2229] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:20:47.035903 7289 raft_consensus.cc:2258] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:47.036072 7288 tablet_replica.cc:330] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401: stopping tablet replica
I20260621 05:20:47.036164 7288 raft_consensus.cc:2229] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:20:47.036259 7288 raft_consensus.cc:2258] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:47.036675 7289 ts_tablet_manager.cc:1905] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:47.036752 7288 ts_tablet_manager.cc:1905] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:47.038352 7289 ts_tablet_manager.cc:1918] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:20:47.038478 7289 log.cc:1196] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/c8567825087340c88cc1184ae9d766ee
I20260621 05:20:47.038830 7289 ts_tablet_manager.cc:1939] T c8567825087340c88cc1184ae9d766ee P 9bbf44c1a24e44d2bbaa6a99fca82bd4: Deleting consensus metadata
I20260621 05:20:47.039366 6787 catalog_manager.cc:4841] TS 9bbf44c1a24e44d2bbaa6a99fca82bd4 (127.2.32.3:45107): tablet c8567825087340c88cc1184ae9d766ee (table test-table [id=9bf1c1056435494bb0860905101b4c4f]) successfully deleted
I20260621 05:20:47.039460 7288 ts_tablet_manager.cc:1918] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:20:47.039547 7288 log.cc:1196] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/c8567825087340c88cc1184ae9d766ee
I20260621 05:20:47.040033 7288 ts_tablet_manager.cc:1939] T c8567825087340c88cc1184ae9d766ee P 266629fc39a24138aec672164860b401: Deleting consensus metadata
W20260621 05:20:47.040225 6785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.32.1:40843 (error 108)
I20260621 05:20:47.040542 6787 catalog_manager.cc:4841] TS 266629fc39a24138aec672164860b401 (127.2.32.2:42155): tablet c8567825087340c88cc1184ae9d766ee (table test-table [id=9bf1c1056435494bb0860905101b4c4f]) successfully deleted
I20260621 05:20:47.040830 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:47.040699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:40843 (local address 127.0.0.1:41154)
0621 05:20:47.040749 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:47.040801 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.1:40843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260621 05:20:47.041020 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 6991
W20260621 05:20:47.041004 6785 catalog_manager.cc:4584] TS 87f9ca0f5f234bac8f06c90a74166705 (127.2.32.1:40843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8567825087340c88cc1184ae9d766ee: Network error: Client connection negotiation failed: client connection to 127.2.32.1:40843: connect: Connection refused (error 111)
I20260621 05:20:47.047430 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 7122
I20260621 05:20:47.052608 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 6769
2026-06-21T05:20:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (631 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260621 05:20:47.074072 2176 mini_oidc.cc:152] Starting JWKS server
I20260621 05:20:47.075525 2176 webserver.cc:460] Webserver started at https://127.0.0.1:44177/ using document root <none> and password file <none>
I20260621 05:20:47.075618 2176 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 05:20:47.075824 2176 webserver.cc:460] Webserver started at http://127.0.0.1:45425/ using document root <none> and password file <none>
2026-06-21T05:20:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:47Z Disabled control of system clock
I20260621 05:20:47.085462 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:40019
--webserver_interface=127.2.32.62
--webserver_port=0
--builtin_ntp_servers=127.2.32.20:33145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:40019
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44177/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782019225111293-2176-0/testchainca.cert with env {}
W20260621 05:20:47.169934 7297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:47.170184 7297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:47.170222 7297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:47.171442 7297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:47.171494 7297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:47.171517 7297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:47.171552 7297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:47.171579 7297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 05:20:47.171612 7297 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 05:20:47.171644 7297 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44177/jwks.json
I20260621 05:20:47.173439 7297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:33145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:40019
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:40019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44177/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782019225111293-2176-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:47.173725 7297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:47.173944 7297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:47.179865 7303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:47.179873 7302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:47.179914 7305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:47.180192 7297 server_base.cc:1034] running on GCE node
I20260621 05:20:47.180424 7297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:47.180709 7297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:47.181881 7297 hybrid_clock.cc:648] HybridClock initialized: now 1782019247181838 us; error 61 us; skew 500 ppm
I20260621 05:20:47.183117 7297 webserver.cc:460] Webserver started at http://127.2.32.62:42295/ using document root <none> and password file <none>
I20260621 05:20:47.183390 7297 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:47.183459 7297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:47.183597 7297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:47.184609 7297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "7ee9414bb050489d838a908ce0c2cccb"
format_stamp: "Formatted at 2026-06-21 05:20:47 on dist-test-slave-b357"
server_key: "599fa0debf50c1244538be3b2c2768ee"
server_key_iv: "d418c93b0c58318365ebdd98b4880ee3"
server_key_version: "encryptionkey@0"
I20260621 05:20:47.184965 7297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "7ee9414bb050489d838a908ce0c2cccb"
format_stamp: "Formatted at 2026-06-21 05:20:47 on dist-test-slave-b357"
server_key: "599fa0debf50c1244538be3b2c2768ee"
server_key_iv: "d418c93b0c58318365ebdd98b4880ee3"
server_key_version: "encryptionkey@0"
I20260621 05:20:47.186400 7297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:20:47.187463 7311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:47.187701 7297 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:47.187803 7297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "7ee9414bb050489d838a908ce0c2cccb"
format_stamp: "Formatted at 2026-06-21 05:20:47 on dist-test-slave-b357"
server_key: "599fa0debf50c1244538be3b2c2768ee"
server_key_iv: "d418c93b0c58318365ebdd98b4880ee3"
server_key_version: "encryptionkey@0"
I20260621 05:20:47.187888 7297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:47.216269 7297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:47.216704 7297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:47.216905 7297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:47.222623 7297 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:40019
I20260621 05:20:47.222667 7363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:40019 every 8 connection(s)
I20260621 05:20:47.223039 7297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:47.224092 7364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:47.226279 7364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb: Bootstrap starting.
I20260621 05:20:47.226848 7364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:47.227149 7364 log.cc:824] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:47.227860 7364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb: No bootstrap required, opened a new log
I20260621 05:20:47.229364 7364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee9414bb050489d838a908ce0c2cccb" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40019 } }
I20260621 05:20:47.229493 7364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:47.229516 7364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ee9414bb050489d838a908ce0c2cccb, State: Initialized, Role: FOLLOWER
I20260621 05:20:47.229622 7364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee9414bb050489d838a908ce0c2cccb" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40019 } }
I20260621 05:20:47.229734 7364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:47.229799 7364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:47.229854 7364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:47.230389 7364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee9414bb050489d838a908ce0c2cccb" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40019 } }
I20260621 05:20:47.230508 7364 leader_election.cc:302] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ee9414bb050489d838a908ce0c2cccb; no voters:
I20260621 05:20:47.230525 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 7297
I20260621 05:20:47.230618 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:20:47.230672 7364 leader_election.cc:288] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:47.230823 2176 external_mini_cluster.cc:1428] Setting key 73b58af4957aeb0e6f129411060d42c4
I20260621 05:20:47.230899 7368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:47.230917 7364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:47.231189 7368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb [term 1 LEADER]: Becoming Leader. State: Replica: 7ee9414bb050489d838a908ce0c2cccb, State: Running, Role: LEADER
I20260621 05:20:47.231343 7368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee9414bb050489d838a908ce0c2cccb" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40019 } }
I20260621 05:20:47.232123 7369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ee9414bb050489d838a908ce0c2cccb. Latest consensus state: current_term: 1 leader_uuid: "7ee9414bb050489d838a908ce0c2cccb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee9414bb050489d838a908ce0c2cccb" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40019 } } }
I20260621 05:20:47.232215 7369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:47.232517 7383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:47.232539 7367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ee9414bb050489d838a908ce0c2cccb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee9414bb050489d838a908ce0c2cccb" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40019 } } }
I20260621 05:20:47.232722 7367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:47.233227 7383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:47.234645 7383 catalog_manager.cc:1347] Generated new cluster ID: ab624765a2c74007aae01c15fb54eed7
I20260621 05:20:47.234700 7383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:47.237612 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:47.231319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60872 (local address 127.2.32.62:40019)
0621 05:20:47.231600 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:47.231608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:47.231644 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:20:47.231773 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:47.231781 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:47.231863 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:47.232361 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:47.232488 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.234244 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.235504 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.236135 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.236263 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:47.236922 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:47.236926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:47.236926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:47.237145 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:47.237187 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:47.237201 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:47.237270 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:47.237338 (+ 68us) server_negotiation.cc:299] Negotiation successful
0621 05:20:47.237414 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":66,"threads_started":1}
I20260621 05:20:47.240358 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:20:47.244814 7383 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:47.245450 7383 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:47.251969 7383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ee9414bb050489d838a908ce0c2cccb: Generated new TSK 0
I20260621 05:20:47.252305 7383 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260621 05:20:47.280591 7394 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260621 05:20:47.280737 7394 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260621 05:20:47.281216 7393 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:47.264756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40019 (local address 127.0.0.1:60886)
0621 05:20:47.265165 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:47.265178 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:47.265272 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:47.265498 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:47.265498 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:47.265501 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0621 05:20:47.265580 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:47.265583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.266554 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.266555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:47.267263 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:47.267265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.267728 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.267732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:47.267786 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:47.267803 (+ 17us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 05:20:47.280818 (+ 13015us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0621 05:20:47.280905 (+ 87us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.32.62:40019: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":251,"threads_started":1}
I20260621 05:20:47.281486 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:47.264986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.2.32.62:40019)
0621 05:20:47.265052 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:47.265054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:47.265267 (+ 213us) server_negotiation.cc:407] Connection header received
0621 05:20:47.265361 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:47.265362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:47.265375 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:47.265467 (+ 92us) server_negotiation.cc:226] Negotiated authn=JWT
0621 05:20:47.265658 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.266440 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.267339 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.267634 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.267709 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:47.267910 (+ 201us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 05:20:47.280683 (+ 12773us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0621 05:20:47.281423 (+ 740us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60886: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":38}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.32.62:40019: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
W20260621 05:20:47.281531 7371 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60886: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
I20260621 05:20:47.281562 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 7297
2026-06-21T05:20:47Z chronyd exiting
I20260621 05:20:47.316823 2176 test_util.cc:175] -----------------------------------------------
I20260621 05:20:47.316919 2176 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782019225111293-2176-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (243 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260621 05:20:47.317600 2176 mini_oidc.cc:152] Starting JWKS server
I20260621 05:20:47.318290 2176 webserver.cc:460] Webserver started at https://127.0.0.1:45561/ using document root <none> and password file <none>
I20260621 05:20:47.318357 2176 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 05:20:47.318511 2176 webserver.cc:460] Webserver started at http://127.0.0.1:38251/ using document root <none> and password file <none>
2026-06-21T05:20:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:47Z Disabled control of system clock
I20260621 05:20:47.326380 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:36093
--webserver_interface=127.2.32.62
--webserver_port=0
--builtin_ntp_servers=127.2.32.20:36387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:36093
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45561/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782019225111293-2176-0/testchainca.cert with env {}
W20260621 05:20:47.405920 7400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:47.406139 7400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:47.406164 7400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:47.407366 7400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:47.407406 7400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:47.407420 7400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:47.407433 7400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:47.407445 7400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 05:20:47.407459 7400 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 05:20:47.407469 7400 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45561/jwks.json
I20260621 05:20:47.409211 7400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:36387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:36093
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:36093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45561/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782019225111293-2176-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:47.409499 7400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:47.409746 7400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:47.414870 7405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:47.414882 7406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:47.415171 7400 server_base.cc:1034] running on GCE node
W20260621 05:20:47.414911 7408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:47.415535 7400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:47.415853 7400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:47.417060 7400 hybrid_clock.cc:648] HybridClock initialized: now 1782019247417016 us; error 64 us; skew 500 ppm
I20260621 05:20:47.418860 7400 webserver.cc:460] Webserver started at http://127.2.32.62:40451/ using document root <none> and password file <none>
I20260621 05:20:47.419224 7400 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:47.419310 7400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:47.419466 7400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:47.420816 7400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "268fbff4fc0a4983841dc26d68d20702"
format_stamp: "Formatted at 2026-06-21 05:20:47 on dist-test-slave-b357"
server_key: "009cbfd3ede662648679ef94530cb89b"
server_key_iv: "05d86e0cb2a8bb11aba3d759a4f2c420"
server_key_version: "encryptionkey@0"
I20260621 05:20:47.421260 7400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "268fbff4fc0a4983841dc26d68d20702"
format_stamp: "Formatted at 2026-06-21 05:20:47 on dist-test-slave-b357"
server_key: "009cbfd3ede662648679ef94530cb89b"
server_key_iv: "05d86e0cb2a8bb11aba3d759a4f2c420"
server_key_version: "encryptionkey@0"
I20260621 05:20:47.422868 7400 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:20:47.423755 7414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:47.424094 7400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:20:47.424181 7400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "268fbff4fc0a4983841dc26d68d20702"
format_stamp: "Formatted at 2026-06-21 05:20:47 on dist-test-slave-b357"
server_key: "009cbfd3ede662648679ef94530cb89b"
server_key_iv: "05d86e0cb2a8bb11aba3d759a4f2c420"
server_key_version: "encryptionkey@0"
I20260621 05:20:47.424232 7400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:47.461210 7400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:47.461745 7400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:47.461908 7400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:47.466241 7400 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:36093
I20260621 05:20:47.466504 7466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:36093 every 8 connection(s)
I20260621 05:20:47.466660 7400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:47.467729 7467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:47.470005 7467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702: Bootstrap starting.
I20260621 05:20:47.470570 7467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:47.470860 7467 log.cc:824] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:47.471683 7467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702: No bootstrap required, opened a new log
I20260621 05:20:47.472131 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 7400
I20260621 05:20:47.472249 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:20:47.472441 2176 external_mini_cluster.cc:1428] Setting key 2ab695f9c7cc484eac53c5be792692b1
I20260621 05:20:47.473980 7467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268fbff4fc0a4983841dc26d68d20702" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36093 } }
I20260621 05:20:47.474152 7467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:47.474190 7467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 268fbff4fc0a4983841dc26d68d20702, State: Initialized, Role: FOLLOWER
I20260621 05:20:47.474311 7467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268fbff4fc0a4983841dc26d68d20702" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36093 } }
I20260621 05:20:47.474386 7467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:47.474408 7467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:47.474473 7467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:47.475226 7467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268fbff4fc0a4983841dc26d68d20702" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36093 } }
I20260621 05:20:47.475332 7467 leader_election.cc:302] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 268fbff4fc0a4983841dc26d68d20702; no voters:
I20260621 05:20:47.475469 7467 leader_election.cc:288] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:47.475541 7472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:47.475692 7472 raft_consensus.cc:686] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702 [term 1 LEADER]: Becoming Leader. State: Replica: 268fbff4fc0a4983841dc26d68d20702, State: Running, Role: LEADER
I20260621 05:20:47.475704 7467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:47.475776 7472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268fbff4fc0a4983841dc26d68d20702" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36093 } }
I20260621 05:20:47.476089 7472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 268fbff4fc0a4983841dc26d68d20702. Latest consensus state: current_term: 1 leader_uuid: "268fbff4fc0a4983841dc26d68d20702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268fbff4fc0a4983841dc26d68d20702" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36093 } } }
I20260621 05:20:47.476183 7472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:47.476104 7473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "268fbff4fc0a4983841dc26d68d20702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268fbff4fc0a4983841dc26d68d20702" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36093 } } }
I20260621 05:20:47.476285 7473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:47.476581 7479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:47.477602 7479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:47.479310 7479 catalog_manager.cc:1347] Generated new cluster ID: b2ea384c2004498cb5f7d52206f5d575
I20260621 05:20:47.479372 7479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:47.479971 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:47.473281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54094 (local address 127.2.32.62:36093)
0621 05:20:47.473501 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:47.473510 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:47.473544 (+ 34us) server_negotiation.cc:407] Connection header received
0621 05:20:47.473939 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:47.473947 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:47.474023 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:47.474185 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:47.474474 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.476424 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.477529 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.478086 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.478222 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:47.479089 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:47.479096 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:47.479096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:47.479461 (+ 365us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:47.479502 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:47.479518 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:47.479594 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:47.479707 (+ 113us) server_negotiation.cc:299] Negotiation successful
0621 05:20:47.479778 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20260621 05:20:47.480873 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:20:47.492607 7479 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:47.493121 7479 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:47.495471 7479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 268fbff4fc0a4983841dc26d68d20702: Generated new TSK 0
I20260621 05:20:47.495664 7479 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260621 05:20:47.526013 7497 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260621 05:20:47.526103 7497 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 05:20:47.526435 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:47.504112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54100 (local address 127.2.32.62:36093)
0621 05:20:47.504176 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:47.504178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:47.504350 (+ 172us) server_negotiation.cc:407] Connection header received
0621 05:20:47.504381 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:47.504391 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:47.504403 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:47.504456 (+ 53us) server_negotiation.cc:226] Negotiated authn=JWT
0621 05:20:47.504597 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.505075 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.506169 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.506349 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.506415 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:47.506590 (+ 175us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 05:20:47.526190 (+ 19600us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0621 05:20:47.526357 (+ 167us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54100: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":37}
W20260621 05:20:47.526494 7471 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54100: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260621 05:20:47.526507 7496 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:47.504050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:36093 (local address 127.0.0.1:54100)
0621 05:20:47.504233 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:47.504243 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:47.504310 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:47.504473 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:47.504473 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:47.504475 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0621 05:20:47.504534 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:47.504537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.505149 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.505150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:47.506054 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:47.506056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.506430 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.506434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:47.506492 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:47.506498 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 05:20:47.526300 (+ 19802us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0621 05:20:47.526408 (+ 108us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.32.62:36093: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":121,"threads_started":1}
I20260621 05:20:47.526796 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 7400
2026-06-21T05:20:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (247 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260621 05:20:47.564953 2176 mini_oidc.cc:152] Starting JWKS server
I20260621 05:20:47.565910 2176 webserver.cc:460] Webserver started at https://127.0.0.1:44477/ using document root <none> and password file <none>
I20260621 05:20:47.565989 2176 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 05:20:47.566210 2176 webserver.cc:460] Webserver started at http://127.0.0.1:37973/ using document root <none> and password file <none>
2026-06-21T05:20:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:47Z Disabled control of system clock
I20260621 05:20:47.574911 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:35733
--webserver_interface=127.2.32.62
--webserver_port=0
--builtin_ntp_servers=127.2.32.20:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:35733
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44477/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260621 05:20:47.654897 7503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:47.655093 7503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:47.655114 7503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:47.656181 7503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:47.656217 7503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:47.656230 7503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:47.656243 7503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:47.656255 7503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 05:20:47.656267 7503 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 05:20:47.656279 7503 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44477/jwks.json
I20260621 05:20:47.658018 7503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:35733
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:35733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44477/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:47.658206 7503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:47.658771 7503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:47.664486 7508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:47.664458 7509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:47.664449 7511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:47.664913 7503 server_base.cc:1034] running on GCE node
I20260621 05:20:47.665199 7503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:47.665457 7503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:47.666647 7503 hybrid_clock.cc:648] HybridClock initialized: now 1782019247666641 us; error 37 us; skew 500 ppm
I20260621 05:20:47.668437 7503 webserver.cc:460] Webserver started at http://127.2.32.62:37417/ using document root <none> and password file <none>
I20260621 05:20:47.668839 7503 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:47.668962 7503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:47.669140 7503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:47.670554 7503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "c4afb00ccaf04a1996ff6470558188e9"
format_stamp: "Formatted at 2026-06-21 05:20:47 on dist-test-slave-b357"
server_key: "46a902654f4c201899edb280c311853c"
server_key_iv: "2d3121735e9b25be246f93939dc4832a"
server_key_version: "encryptionkey@0"
I20260621 05:20:47.671020 7503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "c4afb00ccaf04a1996ff6470558188e9"
format_stamp: "Formatted at 2026-06-21 05:20:47 on dist-test-slave-b357"
server_key: "46a902654f4c201899edb280c311853c"
server_key_iv: "2d3121735e9b25be246f93939dc4832a"
server_key_version: "encryptionkey@0"
I20260621 05:20:47.672751 7503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:47.673650 7517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:47.674062 7503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:47.674182 7503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "c4afb00ccaf04a1996ff6470558188e9"
format_stamp: "Formatted at 2026-06-21 05:20:47 on dist-test-slave-b357"
server_key: "46a902654f4c201899edb280c311853c"
server_key_iv: "2d3121735e9b25be246f93939dc4832a"
server_key_version: "encryptionkey@0"
I20260621 05:20:47.674275 7503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:47.686542 7503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:47.686863 7503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:47.687022 7503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:47.690701 7503 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:35733
I20260621 05:20:47.690910 7569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:35733 every 8 connection(s)
I20260621 05:20:47.691236 7503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:47.691960 7570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:47.695305 7570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9: Bootstrap starting.
I20260621 05:20:47.696012 7570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:47.696302 7570 log.cc:824] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:47.697256 7570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9: No bootstrap required, opened a new log
I20260621 05:20:47.699019 7570 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4afb00ccaf04a1996ff6470558188e9" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 35733 } }
I20260621 05:20:47.699249 7570 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:47.699280 7570 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4afb00ccaf04a1996ff6470558188e9, State: Initialized, Role: FOLLOWER
I20260621 05:20:47.699415 7570 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4afb00ccaf04a1996ff6470558188e9" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 35733 } }
I20260621 05:20:47.699522 7570 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:47.699546 7570 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:47.699594 7570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:47.699689 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 7503
I20260621 05:20:47.699826 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:20:47.700002 2176 external_mini_cluster.cc:1428] Setting key 6c83284f65660a32b3c798aae93baf16
I20260621 05:20:47.700469 7570 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4afb00ccaf04a1996ff6470558188e9" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 35733 } }
I20260621 05:20:47.700587 7570 leader_election.cc:302] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4afb00ccaf04a1996ff6470558188e9; no voters:
I20260621 05:20:47.700747 7570 leader_election.cc:288] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:47.700877 7575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:47.700994 7570 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:47.701012 7575 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9 [term 1 LEADER]: Becoming Leader. State: Replica: c4afb00ccaf04a1996ff6470558188e9, State: Running, Role: LEADER
I20260621 05:20:47.701095 7575 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4afb00ccaf04a1996ff6470558188e9" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 35733 } }
I20260621 05:20:47.701371 7575 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4afb00ccaf04a1996ff6470558188e9. Latest consensus state: current_term: 1 leader_uuid: "c4afb00ccaf04a1996ff6470558188e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4afb00ccaf04a1996ff6470558188e9" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 35733 } } }
I20260621 05:20:47.701452 7575 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:47.701764 7577 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4afb00ccaf04a1996ff6470558188e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4afb00ccaf04a1996ff6470558188e9" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 35733 } } }
I20260621 05:20:47.701903 7577 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:47.702435 7581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:47.703058 7581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:47.704617 7581 catalog_manager.cc:1347] Generated new cluster ID: 1dad4b7cff6e4c6d9458aa3cd55c841d
I20260621 05:20:47.704671 7581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:47.706120 7574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:47.700537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.2.32.62:35733)
0621 05:20:47.700825 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:47.700835 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:47.700871 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:20:47.700981 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:47.700988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:47.701081 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:47.701487 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:47.701592 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.703024 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.704064 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.704524 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.704954 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:47.705366 (+ 412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:47.705369 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:47.705369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:47.705566 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:47.705602 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:47.705614 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:47.705831 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:47.705859 (+ 28us) server_negotiation.cc:299] Negotiation successful
0621 05:20:47.705926 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":194,"threads_started":1}
I20260621 05:20:47.706944 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:20:47.716776 7581 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:47.717689 7581 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:47.723902 7581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4afb00ccaf04a1996ff6470558188e9: Generated new TSK 0
I20260621 05:20:47.724124 7581 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:20:47.739025 7574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:47.735182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52504 (local address 127.2.32.62:35733)
0621 05:20:47.735249 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:47.735252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:47.735384 (+ 132us) server_negotiation.cc:407] Connection header received
0621 05:20:47.735423 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:47.735424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:47.735444 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:47.735513 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0621 05:20:47.735701 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.736194 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.737368 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.737551 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.737619 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:47.737830 (+ 211us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 05:20:47.738823 (+ 993us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44477/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0621 05:20:47.738952 (+ 129us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52504: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44477/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":27}
W20260621 05:20:47.739085 7574 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52504: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44477/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260621 05:20:47.739404 7599 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:47.735013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:35733 (local address 127.0.0.1:52504)
0621 05:20:47.735240 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:47.735254 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:47.735335 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:47.735529 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:47.735530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:47.735533 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0621 05:20:47.735599 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:47.735602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.736276 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.736277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:47.737267 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:47.737269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.737630 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.737638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:47.737694 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:47.737700 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 05:20:47.738923 (+ 1223us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44477/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0621 05:20:47.739023 (+ 100us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.32.62:35733: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44477/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":139,"threads_started":1}
I20260621 05:20:47.739647 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 7503
E20260621 05:20:47.749902 7600 webserver.cc:533] Webserver: sslize failed:
E20260621 05:20:47.749985 7600 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-21T05:20:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (212 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-21T05:20:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:47Z Disabled control of system clock
I20260621 05:20:47.784889 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:45843
--webserver_interface=127.2.32.62
--webserver_port=0
--builtin_ntp_servers=127.2.32.20:42711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:45843
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782019225111293-2176-0/testchainca.cert with env {}
W20260621 05:20:47.869906 7604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:47.870153 7604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:47.870178 7604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:47.871376 7604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:47.871413 7604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:47.871426 7604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:47.871439 7604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:47.871452 7604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 05:20:47.871465 7604 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 05:20:47.871476 7604 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260621 05:20:47.873121 7604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:42711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:45843
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:45843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782019225111293-2176-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:47.873340 7604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:47.873530 7604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:47.879251 7610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:47.879243 7612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:47.879403 7604 server_base.cc:1034] running on GCE node
W20260621 05:20:47.879243 7609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:47.879740 7604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:47.880287 7604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:47.881592 7604 hybrid_clock.cc:648] HybridClock initialized: now 1782019247881569 us; error 37 us; skew 500 ppm
I20260621 05:20:47.883003 7604 webserver.cc:460] Webserver started at http://127.2.32.62:34607/ using document root <none> and password file <none>
I20260621 05:20:47.883316 7604 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:47.883391 7604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:47.883529 7604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:47.884625 7604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "a1864dcfa3584522befede7436eb09e8"
format_stamp: "Formatted at 2026-06-21 05:20:47 on dist-test-slave-b357"
server_key: "e529779e4d4c66f8e9eda0db63f23761"
server_key_iv: "775e8a520f24f81b29288dfc3be1cae7"
server_key_version: "encryptionkey@0"
I20260621 05:20:47.884999 7604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "a1864dcfa3584522befede7436eb09e8"
format_stamp: "Formatted at 2026-06-21 05:20:47 on dist-test-slave-b357"
server_key: "e529779e4d4c66f8e9eda0db63f23761"
server_key_iv: "775e8a520f24f81b29288dfc3be1cae7"
server_key_version: "encryptionkey@0"
I20260621 05:20:47.886475 7604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:20:47.887387 7618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:47.887603 7604 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:47.887720 7604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "a1864dcfa3584522befede7436eb09e8"
format_stamp: "Formatted at 2026-06-21 05:20:47 on dist-test-slave-b357"
server_key: "e529779e4d4c66f8e9eda0db63f23761"
server_key_iv: "775e8a520f24f81b29288dfc3be1cae7"
server_key_version: "encryptionkey@0"
I20260621 05:20:47.887806 7604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:47.897540 7604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:47.897815 7604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:47.897959 7604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:47.903070 7604 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:45843
I20260621 05:20:47.903223 7670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:45843 every 8 connection(s)
I20260621 05:20:47.903514 7604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:47.904446 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:47.906579 7671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8: Bootstrap starting.
I20260621 05:20:47.907088 7671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:47.907406 7671 log.cc:824] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:47.908059 7671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8: No bootstrap required, opened a new log
I20260621 05:20:47.909551 7671 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1864dcfa3584522befede7436eb09e8" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45843 } }
I20260621 05:20:47.909696 7671 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:47.909719 7671 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1864dcfa3584522befede7436eb09e8, State: Initialized, Role: FOLLOWER
I20260621 05:20:47.909852 7671 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1864dcfa3584522befede7436eb09e8" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45843 } }
I20260621 05:20:47.909926 7671 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:47.909958 7671 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:47.909986 7671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:47.910198 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 7604
I20260621 05:20:47.910310 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:20:47.910513 2176 external_mini_cluster.cc:1428] Setting key cf035db467664cd2c3c78af149d81d4b
I20260621 05:20:47.910705 7671 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1864dcfa3584522befede7436eb09e8" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45843 } }
I20260621 05:20:47.910817 7671 leader_election.cc:302] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1864dcfa3584522befede7436eb09e8; no voters:
I20260621 05:20:47.910972 7671 leader_election.cc:288] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:47.911033 7675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:47.911168 7675 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8 [term 1 LEADER]: Becoming Leader. State: Replica: a1864dcfa3584522befede7436eb09e8, State: Running, Role: LEADER
I20260621 05:20:47.911255 7671 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:47.911235 7675 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1864dcfa3584522befede7436eb09e8" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45843 } }
I20260621 05:20:47.911533 7678 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1864dcfa3584522befede7436eb09e8. Latest consensus state: current_term: 1 leader_uuid: "a1864dcfa3584522befede7436eb09e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1864dcfa3584522befede7436eb09e8" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45843 } } }
I20260621 05:20:47.911608 7678 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:47.911733 7676 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1864dcfa3584522befede7436eb09e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1864dcfa3584522befede7436eb09e8" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45843 } } }
I20260621 05:20:47.911868 7676 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:47.912362 7680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:47.912876 7680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:47.914976 7680 catalog_manager.cc:1347] Generated new cluster ID: 2dec3d5359ed4bf2a2df0c225a2600be
I20260621 05:20:47.915035 7680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:20:47.917593 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:47.910987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54614 (local address 127.2.32.62:45843)
0621 05:20:47.911419 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:47.911426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:47.911448 (+ 22us) server_negotiation.cc:407] Connection header received
0621 05:20:47.911579 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:47.911588 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:47.911660 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:47.912037 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:47.912138 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.913482 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.914929 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.915589 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.915715 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:47.916848 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:47.916851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:47.916852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:20:47.917063 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:47.917104 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:20:47.917118 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:47.917200 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:47.917288 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 05:20:47.917366 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":108,"threads_started":1}
I20260621 05:20:47.918856 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:20:47.939610 7680 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:47.940141 7680 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:47.944415 7680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1864dcfa3584522befede7436eb09e8: Generated new TSK 0
I20260621 05:20:47.944588 7680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:20:48.039749 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:47.949687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54616 (local address 127.2.32.62:45843)
0621 05:20:47.949756 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:47.949759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:47.949884 (+ 125us) server_negotiation.cc:407] Connection header received
0621 05:20:47.949989 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:47.949990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:47.950004 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:47.950084 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0621 05:20:47.950233 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.950763 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.951664 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.951887 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.951973 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:47.952169 (+ 196us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 05:20:48.039419 (+ 87250us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0621 05:20:48.039651 (+ 232us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54616: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":44}
W20260621 05:20:48.039832 7677 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54616: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260621 05:20:48.039839 7700 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:47.949605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45843 (local address 127.0.0.1:54616)
0621 05:20:47.949808 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:47.949818 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:47.949903 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:47.950084 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:47.950085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:47.950087 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0621 05:20:47.950148 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:47.950151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.950875 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.950876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:47.951556 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:47.951558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:47.951973 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:47.951981 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:47.952040 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:47.952046 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 05:20:48.039583 (+ 87537us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0621 05:20:48.039729 (+ 146us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.32.62:45843: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":122,"threads_started":1}
I20260621 05:20:48.040172 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 7604
2026-06-21T05:20:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (282 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (90 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (94 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:48 dist-test-slave-b357 krb5kdc[7712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:48 dist-test-slave-b357 krb5kdc[7712](info): set up 2 sockets
Jun 21 05:20:48 dist-test-slave-b357 krb5kdc[7712](info): commencing operation
krb5kdc: starting...
W20260621 05:20:49.283746 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:20:49 dist-test-slave-b357 krb5kdc[7712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:20:49.406488 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:36195
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:33695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:36195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:49.491941 7725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:49.492142 7725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:49.492165 7725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:49.493251 7725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:49.493284 7725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:49.493299 7725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:49.493311 7725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:49.493323 7725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:49.494649 7725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:33695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:36195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:36195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:49.495075 7725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:49.495401 7725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:49.501258 7731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:49.501375 7725 server_base.cc:1034] running on GCE node
W20260621 05:20:49.501396 7730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:49.501396 7733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:49.501899 7725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:49.502240 7725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:49.503412 7725 hybrid_clock.cc:648] HybridClock initialized: now 1782019249503399 us; error 32 us; skew 500 ppm
Jun 21 05:20:49 dist-test-slave-b357 krb5kdc[7712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019249, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:49.511655 7725 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:20:49.512187 7725 webserver.cc:460] Webserver started at http://127.2.32.62:38523/ using document root <none> and password file <none>
I20260621 05:20:49.512490 7725 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:49.512544 7725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:49.512686 7725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:49.513923 7725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "8d0e635fd21242ac992cf86d737ed59e"
format_stamp: "Formatted at 2026-06-21 05:20:49 on dist-test-slave-b357"
server_key: "4821bc22198ec9a158ada84af5a8538e"
server_key_iv: "711f384c944c12c1543f102e890d42d3"
server_key_version: "encryptionkey@0"
I20260621 05:20:49.514328 7725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "8d0e635fd21242ac992cf86d737ed59e"
format_stamp: "Formatted at 2026-06-21 05:20:49 on dist-test-slave-b357"
server_key: "4821bc22198ec9a158ada84af5a8538e"
server_key_iv: "711f384c944c12c1543f102e890d42d3"
server_key_version: "encryptionkey@0"
I20260621 05:20:49.516042 7725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:20:49.517041 7740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:49.517297 7725 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:49.517383 7725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "8d0e635fd21242ac992cf86d737ed59e"
format_stamp: "Formatted at 2026-06-21 05:20:49 on dist-test-slave-b357"
server_key: "4821bc22198ec9a158ada84af5a8538e"
server_key_iv: "711f384c944c12c1543f102e890d42d3"
server_key_version: "encryptionkey@0"
I20260621 05:20:49.517506 7725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:49.538863 7725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:49.541083 7725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:49.541198 7725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:49.545961 7725 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:36195
I20260621 05:20:49.545997 7792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:36195 every 8 connection(s)
I20260621 05:20:49.546360 7725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:49.547576 7793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:49.550294 7793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e: Bootstrap starting.
I20260621 05:20:49.550989 7793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:49.551362 7793 log.cc:824] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:49.552243 7793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e: No bootstrap required, opened a new log
I20260621 05:20:49.552803 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 7725
I20260621 05:20:49.552947 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:20:49.553243 2176 external_mini_cluster.cc:1428] Setting key 620b960833a4e38b72878260df8279a4
I20260621 05:20:49.554788 7793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0e635fd21242ac992cf86d737ed59e" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36195 } }
I20260621 05:20:49.554991 7793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:49.555033 7793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d0e635fd21242ac992cf86d737ed59e, State: Initialized, Role: FOLLOWER
I20260621 05:20:49.555191 7793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0e635fd21242ac992cf86d737ed59e" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36195 } }
I20260621 05:20:49.555281 7793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:49.555305 7793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:49.555363 7793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:49.556658 7793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0e635fd21242ac992cf86d737ed59e" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36195 } }
I20260621 05:20:49.556795 7793 leader_election.cc:302] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d0e635fd21242ac992cf86d737ed59e; no voters:
I20260621 05:20:49.556988 7793 leader_election.cc:288] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:49.557088 7798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:49.557278 7798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e [term 1 LEADER]: Becoming Leader. State: Replica: 8d0e635fd21242ac992cf86d737ed59e, State: Running, Role: LEADER
I20260621 05:20:49.557421 7793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:49.557385 7798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0e635fd21242ac992cf86d737ed59e" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36195 } }
I20260621 05:20:49.557832 7799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d0e635fd21242ac992cf86d737ed59e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0e635fd21242ac992cf86d737ed59e" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36195 } } }
I20260621 05:20:49.557920 7799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:49.558341 7800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d0e635fd21242ac992cf86d737ed59e. Latest consensus state: current_term: 1 leader_uuid: "8d0e635fd21242ac992cf86d737ed59e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0e635fd21242ac992cf86d737ed59e" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36195 } } }
I20260621 05:20:49.558419 7804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:49.558440 7800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:49.559207 7804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:49.560909 7804 catalog_manager.cc:1347] Generated new cluster ID: a4010880a97e4daebf0228425c5e4856
I20260621 05:20:49.560963 7804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:20:49 dist-test-slave-b357 krb5kdc[7712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:49.565508 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:49.553960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42106 (local address 127.2.32.62:36195)
0621 05:20:49.554216 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:49.554228 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:49.554289 (+ 61us) server_negotiation.cc:407] Connection header received
0621 05:20:49.554728 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:49.554738 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:49.554805 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:49.554988 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:49.555910 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:49.557550 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:49.558844 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:49.559590 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:49.559738 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:49.562349 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:49.562352 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:49.562353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:49.562364 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:49.564022 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:49.564393 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:49.564395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:49.564396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:49.564495 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:49.564708 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:49.564708 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:49.564708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:49.564951 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:49.565132 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:49.565194 (+ 62us) server_negotiation.cc:299] Negotiation successful
0621 05:20:49.565329 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":111,"threads_started":1}
I20260621 05:20:49.569659 7804 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:49.570173 7804 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:49.576546 7804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d0e635fd21242ac992cf86d737ed59e: Generated new TSK 0
I20260621 05:20:49.576725 7804 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
I20260621 05:20:49.610599 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:36195
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:33695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:49.705631 7819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:49.705917 7819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:49.705947 7819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:49.707361 7819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:49.707415 7819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:49.707448 7819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:20:49.709030 7819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:33695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:36195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:49.709288 7819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:49.709517 7819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:49.715349 7825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:49.715525 7827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:49.715543 7824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:49.715768 7819 server_base.cc:1034] running on GCE node
I20260621 05:20:49.716020 7819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:49.716286 7819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:49.717456 7819 hybrid_clock.cc:648] HybridClock initialized: now 1782019249717440 us; error 43 us; skew 500 ppm
Jun 21 05:20:49 dist-test-slave-b357 krb5kdc[7712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019249, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:49.720352 7819 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:20:49.720906 7819 webserver.cc:460] Webserver started at http://127.2.32.1:36841/ using document root <none> and password file <none>
I20260621 05:20:49.721323 7819 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:49.721415 7819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:49.721588 7819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:49.722934 7819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "2cf6d04b903f4e76a95d011b7eb9ed02"
format_stamp: "Formatted at 2026-06-21 05:20:49 on dist-test-slave-b357"
server_key: "8cb7d57e595bbd75bf01bc0bb4157279"
server_key_iv: "e02b457345513efe835f564b9db553b3"
server_key_version: "encryptionkey@0"
I20260621 05:20:49.723431 7819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "2cf6d04b903f4e76a95d011b7eb9ed02"
format_stamp: "Formatted at 2026-06-21 05:20:49 on dist-test-slave-b357"
server_key: "8cb7d57e595bbd75bf01bc0bb4157279"
server_key_iv: "e02b457345513efe835f564b9db553b3"
server_key_version: "encryptionkey@0"
I20260621 05:20:49.725055 7819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:49.726102 7834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:49.726444 7819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:49.726579 7819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "2cf6d04b903f4e76a95d011b7eb9ed02"
format_stamp: "Formatted at 2026-06-21 05:20:49 on dist-test-slave-b357"
server_key: "8cb7d57e595bbd75bf01bc0bb4157279"
server_key_iv: "e02b457345513efe835f564b9db553b3"
server_key_version: "encryptionkey@0"
I20260621 05:20:49.726680 7819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:49.738768 7819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:49.741317 7819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:49.741477 7819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:49.741724 7819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:49.742146 7819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:49.742187 7819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:49.742213 7819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:49.742226 7819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:49.750000 7819 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:45723
I20260621 05:20:49.750065 7947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:45723 every 8 connection(s)
I20260621 05:20:49.750435 7819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:20:49.757278 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 7819
I20260621 05:20:49.757400 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:20:49.757593 2176 external_mini_cluster.cc:1428] Setting key a69dff547371975f952b96219e3f5853
Jun 21 05:20:49 dist-test-slave-b357 krb5kdc[7712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019249, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:49.762630 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:49.751398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:43715 (local address 127.2.32.62:36195)
0621 05:20:49.751491 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:49.751494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:49.751865 (+ 371us) server_negotiation.cc:407] Connection header received
0621 05:20:49.752772 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:49.752774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:49.752792 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:49.752890 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:49.753949 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:49.754482 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:49.755590 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:49.755792 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:49.755863 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:49.759246 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:49.759247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:49.759248 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:49.759258 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:49.760871 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:49.761416 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:49.761417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:49.761418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:49.761471 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:49.761805 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:49.761806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:49.761807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:49.761947 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:49.762117 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:49.762474 (+ 357us) server_negotiation.cc:299] Negotiation successful
0621 05:20:49.762552 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 05:20:49.762933 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:49.751370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:36195 (local address 127.2.32.1:43715)
0621 05:20:49.751600 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:49.751642 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:49.752631 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:49.753014 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:49.753020 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:49.753494 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:49.753783 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:49.753800 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:49.754601 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:49.754602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:49.755501 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:49.755504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:49.755917 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:49.755919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:49.756068 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:49.756879 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:49.756890 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:49.759089 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:49.760969 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:49.760975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:49.760977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:49.761332 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:49.761557 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:49.761558 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:49.761558 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:49.761704 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:49.762109 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:49.762116 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:49.762287 (+ 171us) client_negotiation.cc:769] Sending connection context
0621 05:20:49.762427 (+ 140us) client_negotiation.cc:240] Negotiation successful
0621 05:20:49.762633 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":114,"threads_started":1}
I20260621 05:20:49.763616 7948 heartbeater.cc:347] Connected to a master server at 127.2.32.62:36195
I20260621 05:20:49.763729 7948 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:49.763926 7948 heartbeater.cc:511] Master 127.2.32.62:36195 requested a full tablet report, sending...
I20260621 05:20:49.764707 7757 ts_manager.cc:194] Registered new tserver with Master: 2cf6d04b903f4e76a95d011b7eb9ed02 (127.2.32.1:45723)
I20260621 05:20:49.765486 7757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:43715
WARNING: no policy specified for kudu/127.2.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
I20260621 05:20:49.786047 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:36195
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:33695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:49.877307 7953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:49.877522 7953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:49.877547 7953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:49.879045 7953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:49.879093 7953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:49.879120 7953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:20:49.880839 7953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:33695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:36195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:49.881093 7953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:49.881356 7953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:49.887157 7959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:49.887310 7961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:49.887169 7958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:49.887409 7953 server_base.cc:1034] running on GCE node
I20260621 05:20:49.887681 7953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:49.887925 7953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:49.889132 7953 hybrid_clock.cc:648] HybridClock initialized: now 1782019249889122 us; error 63 us; skew 500 ppm
Jun 21 05:20:49 dist-test-slave-b357 krb5kdc[7712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019249, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:49.891800 7953 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:20:49.892232 7953 webserver.cc:460] Webserver started at http://127.2.32.2:39515/ using document root <none> and password file <none>
I20260621 05:20:49.892452 7953 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:49.892521 7953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:49.892668 7953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:49.894002 7953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "ddbc49fa3f6f4a3f98263f1f4e037010"
format_stamp: "Formatted at 2026-06-21 05:20:49 on dist-test-slave-b357"
server_key: "d0708339721b9d777f1967132bec63f1"
server_key_iv: "199219b01adda730bc2f872fa2ab5a59"
server_key_version: "encryptionkey@0"
I20260621 05:20:49.894508 7953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "ddbc49fa3f6f4a3f98263f1f4e037010"
format_stamp: "Formatted at 2026-06-21 05:20:49 on dist-test-slave-b357"
server_key: "d0708339721b9d777f1967132bec63f1"
server_key_iv: "199219b01adda730bc2f872fa2ab5a59"
server_key_version: "encryptionkey@0"
I20260621 05:20:49.896216 7953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260621 05:20:49.897212 7968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:49.897467 7953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:49.897585 7953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "ddbc49fa3f6f4a3f98263f1f4e037010"
format_stamp: "Formatted at 2026-06-21 05:20:49 on dist-test-slave-b357"
server_key: "d0708339721b9d777f1967132bec63f1"
server_key_iv: "199219b01adda730bc2f872fa2ab5a59"
server_key_version: "encryptionkey@0"
I20260621 05:20:49.897677 7953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:49.928577 7953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:49.931416 7953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:49.931596 7953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:49.931900 7953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:49.932210 7953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:49.932266 7953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:49.932307 7953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:49.932335 7953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:49.938215 7953 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:46373
I20260621 05:20:49.938295 8081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:46373 every 8 connection(s)
I20260621 05:20:49.938616 7953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:20:49.942035 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 7953
I20260621 05:20:49.942160 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:20:49.942366 2176 external_mini_cluster.cc:1428] Setting key fa5aa9135831b75d55334d3901c649db
Jun 21 05:20:49 dist-test-slave-b357 krb5kdc[7712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019249, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:49.950093 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:49.939687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:36195 (local address 127.2.32.2:55669)
0621 05:20:49.939907 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:49.939961 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:49.941098 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:49.941569 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:49.941579 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:49.941859 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:49.942062 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:49.942067 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:49.942929 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:49.942930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:49.943792 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:49.943795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:49.944246 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:49.944248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:49.944405 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:49.944937 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:49.944945 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:49.946278 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:49.948426 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:49.948432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:49.948433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:49.948662 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:49.948984 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:49.948986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:49.948986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:49.949179 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:49.949555 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:49.949561 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:49.949700 (+ 139us) client_negotiation.cc:769] Sending connection context
0621 05:20:49.949800 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 05:20:49.949929 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":107,"threads_started":1}
I20260621 05:20:49.950158 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:49.939741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:55669 (local address 127.2.32.62:36195)
0621 05:20:49.939836 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:49.939839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:49.940079 (+ 240us) server_negotiation.cc:407] Connection header received
0621 05:20:49.941403 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:49.941405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:49.941419 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:49.941480 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:49.942173 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:49.942823 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:49.943898 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:49.944116 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:49.944185 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:49.946367 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:49.946369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:49.946369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:49.946377 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:49.948290 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:49.948751 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:49.948752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:49.948752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:49.948820 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:49.949280 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:49.949281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:49.949281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:49.949423 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:49.949507 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:49.949849 (+ 342us) server_negotiation.cc:299] Negotiation successful
0621 05:20:49.950021 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
WARNING: no policy specified for kudu/127.2.32.3@KRBTEST.COM; defaulting to no policy
I20260621 05:20:49.950872 8082 heartbeater.cc:347] Connected to a master server at 127.2.32.62:36195
I20260621 05:20:49.950964 8082 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:49.951221 8082 heartbeater.cc:511] Master 127.2.32.62:36195 requested a full tablet report, sending...
I20260621 05:20:49.951656 7757 ts_manager.cc:194] Registered new tserver with Master: ddbc49fa3f6f4a3f98263f1f4e037010 (127.2.32.2:46373)
I20260621 05:20:49.952126 7757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:55669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
I20260621 05:20:49.970043 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:36195
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:33695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:50.063468 8087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:50.063724 8087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:50.063769 8087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:50.065577 8087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:50.065634 8087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:50.065686 8087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:20:50.067343 8087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:33695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:36195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:50.067533 8087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:50.067710 8087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:20:50.073258 8087 server_base.cc:1034] running on GCE node
W20260621 05:20:50.073065 8093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:50.073060 8092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:50.073060 8095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:50.073779 8087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:50.074096 8087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:50.075313 8087 hybrid_clock.cc:648] HybridClock initialized: now 1782019250075298 us; error 64 us; skew 500 ppm
Jun 21 05:20:50 dist-test-slave-b357 krb5kdc[7712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019250, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:50.078738 8087 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:20:50.079348 8087 webserver.cc:460] Webserver started at http://127.2.32.3:43907/ using document root <none> and password file <none>
I20260621 05:20:50.079744 8087 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:50.079841 8087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:50.080017 8087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:50.081269 8087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "1ccc6410cb724a2d883335b1a339275c"
format_stamp: "Formatted at 2026-06-21 05:20:50 on dist-test-slave-b357"
server_key: "19ded9369b71f0845e5f6950c55c653d"
server_key_iv: "3ab526aad000395ad133f273457dd5c6"
server_key_version: "encryptionkey@0"
I20260621 05:20:50.081702 8087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "1ccc6410cb724a2d883335b1a339275c"
format_stamp: "Formatted at 2026-06-21 05:20:50 on dist-test-slave-b357"
server_key: "19ded9369b71f0845e5f6950c55c653d"
server_key_iv: "3ab526aad000395ad133f273457dd5c6"
server_key_version: "encryptionkey@0"
I20260621 05:20:50.083356 8087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260621 05:20:50.084398 8102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:50.084638 8087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:50.084741 8087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "1ccc6410cb724a2d883335b1a339275c"
format_stamp: "Formatted at 2026-06-21 05:20:50 on dist-test-slave-b357"
server_key: "19ded9369b71f0845e5f6950c55c653d"
server_key_iv: "3ab526aad000395ad133f273457dd5c6"
server_key_version: "encryptionkey@0"
I20260621 05:20:50.084825 8087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:50.102339 8087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:50.104658 8087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:50.104773 8087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:50.104983 8087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:50.105312 8087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:50.105348 8087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:50.105379 8087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:50.105393 8087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:50.111254 8087 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:40357
I20260621 05:20:50.111311 8215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:40357 every 8 connection(s)
I20260621 05:20:50.111625 8087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:20:50.115084 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 8087
I20260621 05:20:50.115185 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:20:50.115337 2176 external_mini_cluster.cc:1428] Setting key 33f4f31cb15bdaae7475437aef764f17
Jun 21 05:20:50 dist-test-slave-b357 krb5kdc[7712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019250, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:20:50.122995 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:50.112314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:41135 (local address 127.2.32.62:36195)
0621 05:20:50.112371 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:50.112390 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:50.112560 (+ 170us) server_negotiation.cc:407] Connection header received
0621 05:20:50.113938 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:50.113939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:50.113951 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:50.114001 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:50.114622 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:50.115320 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:50.117077 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:50.117265 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:50.117336 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:50.119598 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:50.119599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:50.119599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:50.119605 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:50.121542 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:50.122030 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:50.122032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:50.122032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:50.122095 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:50.122355 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:50.122356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:50.122356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:50.122470 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:50.122545 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:50.122817 (+ 272us) server_negotiation.cc:299] Negotiation successful
0621 05:20:50.122902 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 05:20:50.123214 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:50.112279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:36195 (local address 127.2.32.3:41135)
0621 05:20:50.112453 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:50.112485 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:50.113835 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:50.114075 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:50.114079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:50.114330 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:50.114522 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:50.114526 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:50.115514 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:50.115517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:50.116959 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:50.116964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:50.117379 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:50.117380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:50.117509 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:50.117957 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:50.117964 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:50.119502 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:50.121701 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:50.121707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:50.121710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:50.121936 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:50.122165 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:50.122165 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:50.122166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:50.122277 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:50.122538 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:50.122544 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:50.122684 (+ 140us) client_negotiation.cc:769] Sending connection context
0621 05:20:50.122795 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 05:20:50.122958 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260621 05:20:50.123718 8216 heartbeater.cc:347] Connected to a master server at 127.2.32.62:36195
I20260621 05:20:50.123795 8216 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:50.123970 8216 heartbeater.cc:511] Master 127.2.32.62:36195 requested a full tablet report, sending...
I20260621 05:20:50.124445 7757 ts_manager.cc:194] Registered new tserver with Master: 1ccc6410cb724a2d883335b1a339275c (127.2.32.3:40357)
I20260621 05:20:50.124819 7757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:41135
I20260621 05:20:50.127499 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/test-admin.keytab.
Jun 21 05:20:50 dist-test-slave-b357 krb5kdc[7712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:50.144150 2176 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260621 05:20:50.146191 8223 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:50.145203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36448 (local address 127.2.32.1:45723)
0621 05:20:50.145407 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:50.145413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:50.145424 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:20:50.145455 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:50.145457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:50.145473 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:50.145570 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:50.146061 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36448: BlockingRecv error: recv got EOF from 127.0.0.1:36448 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":78,"threads_started":1}
W20260621 05:20:50.146381 2176 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/krb5cc' of type FILE
Jun 21 05:20:50 dist-test-slave-b357 krb5kdc[7712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:50.147503 2176 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 05:20:50 dist-test-slave-b357 krb5kdc[7712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:50.148655 2176 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 05:20:50 dist-test-slave-b357 krb5kdc[7712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.1@KRBTEST.COM
I20260621 05:20:50.157446 8223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:50.149358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36464 (local address 127.2.32.1:45723)
0621 05:20:50.149423 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:50.149425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:50.149436 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:20:50.149510 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:50.149511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:50.149527 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:50.149605 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:50.150059 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:50.150796 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:50.151640 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:50.151963 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:50.152061 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:50.154462 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:50.154470 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:50.154470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:50.154478 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:50.156210 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:50.156623 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:50.156627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:50.156627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:50.156686 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:50.156910 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:50.156911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:50.156911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:50.157040 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:50.157196 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:50.157246 (+ 50us) server_negotiation.cc:299] Negotiation successful
0621 05:20:50.157324 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260621 05:20:50.159832 8223 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:50.159249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36470 (local address 127.2.32.1:45723)
0621 05:20:50.159316 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:50.159318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:50.159383 (+ 65us) server_negotiation.cc:407] Connection header received
0621 05:20:50.159453 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:50.159454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:50.159465 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:50.159512 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:50.159790 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36470: BlockingRecv error: recv got EOF from 127.0.0.1:36470 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260621 05:20:50.160028 2176 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/krb5cc' of type FILE
Jun 21 05:20:50 dist-test-slave-b357 krb5kdc[7712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:50.161016 2176 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 05:20:50 dist-test-slave-b357 krb5kdc[7712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:50.161952 2176 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 05:20:50 dist-test-slave-b357 krb5kdc[7712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.1@KRBTEST.COM
I20260621 05:20:50.170398 8223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:50.162557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.2.32.1:45723)
0621 05:20:50.162629 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:50.162632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:50.162760 (+ 128us) server_negotiation.cc:407] Connection header received
0621 05:20:50.162876 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:50.162877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:50.162888 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:50.162972 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:50.163595 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:50.164189 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:50.165056 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:50.165291 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:50.165386 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:50.167693 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:50.167695 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:50.167695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:50.167704 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:50.169140 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:50.169566 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:50.169567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:50.169568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:50.169651 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:50.169868 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:50.169869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:50.169870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:50.169977 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:50.170088 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:50.170205 (+ 117us) server_negotiation.cc:299] Negotiation successful
0621 05:20:50.170296 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260621 05:20:50.172272 8223 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:20:50.171600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.2.32.1:45723)
0621 05:20:50.171674 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:50.171677 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:50.171752 (+ 75us) server_negotiation.cc:407] Connection header received
0621 05:20:50.171832 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:50.171833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:50.171843 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:50.171911 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:50.172228 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36496: BlockingRecv error: recv got EOF from 127.0.0.1:36496 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260621 05:20:50.172576 2176 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782019225111293-2176-0/krb5kdc/krb5cc' of type FILE
Jun 21 05:20:50 dist-test-slave-b357 krb5kdc[7712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:50.173575 2176 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 05:20:50 dist-test-slave-b357 krb5kdc[7712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:50.174386 2176 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 05:20:50 dist-test-slave-b357 krb5kdc[7712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.1@KRBTEST.COM
I20260621 05:20:50.182430 8223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:50.174881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36512 (local address 127.2.32.1:45723)
0621 05:20:50.174933 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:50.174935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:50.174962 (+ 27us) server_negotiation.cc:407] Connection header received
0621 05:20:50.175019 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:50.175019 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:50.175026 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:50.175086 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:50.175660 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:50.176151 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:50.176867 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:50.177051 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:50.177122 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:50.179429 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:50.179436 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:50.179436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:50.179442 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:50.181035 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:50.181610 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:50.181611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:50.181611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:50.181676 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:50.181918 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:50.181919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:50.181920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:50.182014 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:50.182122 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:50.182257 (+ 135us) server_negotiation.cc:299] Negotiation successful
0621 05:20:50.182334 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 05:20:50.183070 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 7819
I20260621 05:20:50.187707 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 7953
I20260621 05:20:50.191819 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 8087
I20260621 05:20:50.196943 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 7725
2026-06-21T05:20:50Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1972 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:20:50 dist-test-slave-b357 krb5kdc[8236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:20:50 dist-test-slave-b357 krb5kdc[8236](info): set up 2 sockets
Jun 21 05:20:50 dist-test-slave-b357 krb5kdc[8236](info): commencing operation
krb5kdc: starting...
W20260621 05:20:51.259975 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:20:51 dist-test-slave-b357 krb5kdc[8236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:20:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:20:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.62.keytab.
Entry for principal oryx/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.62.keytab.
I20260621 05:20:51.387555 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:45421
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.62.keytab
--principal=oryx/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:45421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:51.469529 8249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:51.469762 8249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:51.469785 8249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:51.470917 8249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:51.470958 8249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:51.470973 8249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:51.470986 8249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:51.470999 8249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:51.472470 8249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:45421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.62.keytab
--principal=oryx/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:45421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:51.472702 8249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:51.472946 8249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:51.478410 8254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:51.478428 8255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:51.478457 8257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:51.478824 8249 server_base.cc:1034] running on GCE node
I20260621 05:20:51.479063 8249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:51.479383 8249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:51.480603 8249 hybrid_clock.cc:648] HybridClock initialized: now 1782019251480589 us; error 39 us; skew 500 ppm
Jun 21 05:20:51 dist-test-slave-b357 krb5kdc[8236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019251, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:51.483826 8249 init.cc:377] Logged in from keytab as oryx/127.2.32.62@KRBTEST.COM (short username oryx)
I20260621 05:20:51.484398 8249 webserver.cc:460] Webserver started at http://127.2.32.62:34471/ using document root <none> and password file <none>
I20260621 05:20:51.484673 8249 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:51.484746 8249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:51.484907 8249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:51.486155 8249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "087a170fca294b98b94961bab595c623"
format_stamp: "Formatted at 2026-06-21 05:20:51 on dist-test-slave-b357"
server_key: "9f7ce1d62eaab1e79e0c4cd8b262cf7c"
server_key_iv: "eb76ceab45ab3bc8ce7eff5cbb24531f"
server_key_version: "encryptionkey@0"
I20260621 05:20:51.486582 8249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "087a170fca294b98b94961bab595c623"
format_stamp: "Formatted at 2026-06-21 05:20:51 on dist-test-slave-b357"
server_key: "9f7ce1d62eaab1e79e0c4cd8b262cf7c"
server_key_iv: "eb76ceab45ab3bc8ce7eff5cbb24531f"
server_key_version: "encryptionkey@0"
I20260621 05:20:51.488370 8249 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260621 05:20:51.489419 8264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:51.489733 8249 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:51.489874 8249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "087a170fca294b98b94961bab595c623"
format_stamp: "Formatted at 2026-06-21 05:20:51 on dist-test-slave-b357"
server_key: "9f7ce1d62eaab1e79e0c4cd8b262cf7c"
server_key_iv: "eb76ceab45ab3bc8ce7eff5cbb24531f"
server_key_version: "encryptionkey@0"
I20260621 05:20:51.489969 8249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:51.516273 8249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:51.519089 8249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:51.519282 8249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:51.523355 8249 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:45421
I20260621 05:20:51.523442 8316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:45421 every 8 connection(s)
I20260621 05:20:51.523788 8249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:20:51.524901 8317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:51.527689 8317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623: Bootstrap starting.
I20260621 05:20:51.528272 8317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:51.528554 8317 log.cc:824] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:51.529515 8317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623: No bootstrap required, opened a new log
I20260621 05:20:51.531625 8317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } }
I20260621 05:20:51.531987 8317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:51.532020 8317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 087a170fca294b98b94961bab595c623, State: Initialized, Role: FOLLOWER
I20260621 05:20:51.532305 8317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } }
I20260621 05:20:51.532421 8317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:20:51.532454 8317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:20:51.532486 8317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:51.533442 8317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } }
I20260621 05:20:51.533478 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 8249
I20260621 05:20:51.533535 8317 leader_election.cc:302] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 087a170fca294b98b94961bab595c623; no voters:
I20260621 05:20:51.533623 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:20:51.533692 8317 leader_election.cc:288] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:20:51.533728 8320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:51.533843 8320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [term 1 LEADER]: Becoming Leader. State: Replica: 087a170fca294b98b94961bab595c623, State: Running, Role: LEADER
I20260621 05:20:51.533842 2176 external_mini_cluster.cc:1428] Setting key b556cbfc04809bcdb42666f29848e556
I20260621 05:20:51.533972 8317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:51.533927 8320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } }
I20260621 05:20:51.534302 8320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 087a170fca294b98b94961bab595c623. Latest consensus state: current_term: 1 leader_uuid: "087a170fca294b98b94961bab595c623" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } }
I20260621 05:20:51.534402 8320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:51.534312 8321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "087a170fca294b98b94961bab595c623" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } }
I20260621 05:20:51.534569 8321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:51.534632 8327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:20:51.535202 8327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:20:51.537485 8327 catalog_manager.cc:1347] Generated new cluster ID: 0352e39b766e443fb799d4ba4fbb2187
I20260621 05:20:51.537564 8327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:20:51 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM
I20260621 05:20:51.546813 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:51.534523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47988 (local address 127.2.32.62:45421)
0621 05:20:51.534734 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:51.534744 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:51.534910 (+ 166us) server_negotiation.cc:407] Connection header received
0621 05:20:51.535037 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:51.535045 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:51.535101 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:51.535528 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:51.536192 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:51.537977 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:51.539179 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:51.539996 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:51.540111 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:51.543091 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:51.543095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:51.543095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:51.543110 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:51.545161 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:51.545556 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:51.545559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:51.545559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:51.545685 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:51.545892 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:51.545892 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:51.545893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:51.546192 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:51.546335 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:51.546435 (+ 100us) server_negotiation.cc:299] Negotiation successful
0621 05:20:51.546584 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
I20260621 05:20:51.552748 8327 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:20:51.553242 8327 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:20:51.558692 8327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623: Generated new TSK 0
I20260621 05:20:51.558882 8327 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.1.keytab.
Entry for principal oryx/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.1.keytab.
I20260621 05:20:51.589781 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:45421
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.1.keytab
--principal=oryx/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:51.675729 8343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:51.676164 8343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:51.676208 8343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:51.678006 8343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:51.678085 8343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:51.678125 8343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:20:51.680368 8343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.1.keytab
--principal=oryx/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:45421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:51.680958 8343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:51.681308 8343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:51.686823 8348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:51.686811 8349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:51.686811 8351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:51.687103 8343 server_base.cc:1034] running on GCE node
I20260621 05:20:51.687333 8343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:51.687590 8343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:51.688851 8343 hybrid_clock.cc:648] HybridClock initialized: now 1782019251688767 us; error 97 us; skew 500 ppm
Jun 21 05:20:51 dist-test-slave-b357 krb5kdc[8236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019251, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:51.692009 8343 init.cc:377] Logged in from keytab as oryx/127.2.32.1@KRBTEST.COM (short username oryx)
I20260621 05:20:51.692658 8343 webserver.cc:460] Webserver started at http://127.2.32.1:35091/ using document root <none> and password file <none>
I20260621 05:20:51.692924 8343 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:51.692998 8343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:51.693164 8343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:51.694722 8343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "4b7d4becc8c54574807da6bc9cddabf1"
format_stamp: "Formatted at 2026-06-21 05:20:51 on dist-test-slave-b357"
server_key: "b689da97e57cc44b0d144a783c000927"
server_key_iv: "cae37d898fd3ccc8075ed816a946f6f5"
server_key_version: "encryptionkey@0"
I20260621 05:20:51.695263 8343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "4b7d4becc8c54574807da6bc9cddabf1"
format_stamp: "Formatted at 2026-06-21 05:20:51 on dist-test-slave-b357"
server_key: "b689da97e57cc44b0d144a783c000927"
server_key_iv: "cae37d898fd3ccc8075ed816a946f6f5"
server_key_version: "encryptionkey@0"
I20260621 05:20:51.696771 8343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:51.697781 8358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:51.698110 8343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:51.698269 8343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "4b7d4becc8c54574807da6bc9cddabf1"
format_stamp: "Formatted at 2026-06-21 05:20:51 on dist-test-slave-b357"
server_key: "b689da97e57cc44b0d144a783c000927"
server_key_iv: "cae37d898fd3ccc8075ed816a946f6f5"
server_key_version: "encryptionkey@0"
I20260621 05:20:51.698370 8343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:51.719573 8343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:51.722394 8343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:51.722587 8343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:51.722854 8343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:51.723210 8343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:51.723266 8343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:51.723309 8343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:51.723340 8343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:51.730065 8343 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:42265
I20260621 05:20:51.730216 8471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:42265 every 8 connection(s)
I20260621 05:20:51.730496 8343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:20:51.735342 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 8343
I20260621 05:20:51.735428 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:20:51.735632 2176 external_mini_cluster.cc:1428] Setting key 9ca3f0bdcf56ee61273e6052162a230d
Jun 21 05:20:51 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019251, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM
I20260621 05:20:51.741969 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:51.731522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:39837 (local address 127.2.32.62:45421)
0621 05:20:51.731614 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:51.731616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:51.731877 (+ 261us) server_negotiation.cc:407] Connection header received
0621 05:20:51.732514 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:51.732515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:51.732529 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:51.732609 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:51.733785 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:51.734397 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:51.735952 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:51.736213 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:51.736277 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:51.738582 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:51.738584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:51.738584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:51.738592 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:51.740498 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:51.741027 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:51.741028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:51.741028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:51.741084 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:51.741331 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:51.741332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:51.741332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:51.741448 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:51.741526 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:51.741798 (+ 272us) server_negotiation.cc:299] Negotiation successful
0621 05:20:51.741877 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 05:20:51.742189 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:51.731489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:39837)
0621 05:20:51.731707 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:51.731757 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:51.732343 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:51.732678 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:51.732682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:51.733339 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:51.733575 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:51.733580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:51.734518 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:51.734520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:51.735855 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:51.735860 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:51.736299 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:51.736300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:51.736450 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:51.737010 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:51.737019 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:51.738488 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:51.740694 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:51.740701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:51.740703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:51.740924 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:51.741159 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:51.741159 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:51.741160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:51.741263 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:51.741532 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:51.741539 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:51.741669 (+ 130us) client_negotiation.cc:769] Sending connection context
0621 05:20:51.741777 (+ 108us) client_negotiation.cc:240] Negotiation successful
0621 05:20:51.741936 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":133,"threads_started":1}
I20260621 05:20:51.742882 8472 heartbeater.cc:347] Connected to a master server at 127.2.32.62:45421
I20260621 05:20:51.742993 8472 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:51.743222 8472 heartbeater.cc:511] Master 127.2.32.62:45421 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.2.32.2@KRBTEST.COM; defaulting to no policy
I20260621 05:20:51.743821 8281 ts_manager.cc:194] Registered new tserver with Master: 4b7d4becc8c54574807da6bc9cddabf1 (127.2.32.1:42265)
I20260621 05:20:51.744415 8281 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:39837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.2.keytab.
Entry for principal oryx/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.2.keytab.
I20260621 05:20:51.763429 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:45421
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.2.keytab
--principal=oryx/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:51.850039 8477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:51.850299 8477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:51.850330 8477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:51.851729 8477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:51.851788 8477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:51.851820 8477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:20:51.853420 8477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.2.keytab
--principal=oryx/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:45421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:51.853659 8477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:51.853899 8477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:51.859520 8485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:51.859520 8482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:51.859553 8483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:51.859860 8477 server_base.cc:1034] running on GCE node
I20260621 05:20:51.860106 8477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:51.860380 8477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:51.861575 8477 hybrid_clock.cc:648] HybridClock initialized: now 1782019251861557 us; error 30 us; skew 500 ppm
Jun 21 05:20:51 dist-test-slave-b357 krb5kdc[8236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019251, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:51.864256 8477 init.cc:377] Logged in from keytab as oryx/127.2.32.2@KRBTEST.COM (short username oryx)
I20260621 05:20:51.864703 8477 webserver.cc:460] Webserver started at http://127.2.32.2:37883/ using document root <none> and password file <none>
I20260621 05:20:51.864934 8477 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:51.865003 8477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:51.865151 8477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:51.866214 8477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "533996b4b53d4acb81ac269071c0adc5"
format_stamp: "Formatted at 2026-06-21 05:20:51 on dist-test-slave-b357"
server_key: "28562493b8c782540bcaac537cab7813"
server_key_iv: "d73fda16091034022942a9b2f170044d"
server_key_version: "encryptionkey@0"
I20260621 05:20:51.866580 8477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "533996b4b53d4acb81ac269071c0adc5"
format_stamp: "Formatted at 2026-06-21 05:20:51 on dist-test-slave-b357"
server_key: "28562493b8c782540bcaac537cab7813"
server_key_iv: "d73fda16091034022942a9b2f170044d"
server_key_version: "encryptionkey@0"
I20260621 05:20:51.868165 8477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:20:51.869333 8492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:51.869618 8477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:51.869767 8477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "533996b4b53d4acb81ac269071c0adc5"
format_stamp: "Formatted at 2026-06-21 05:20:51 on dist-test-slave-b357"
server_key: "28562493b8c782540bcaac537cab7813"
server_key_iv: "d73fda16091034022942a9b2f170044d"
server_key_version: "encryptionkey@0"
I20260621 05:20:51.869875 8477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:51.885205 8477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:51.891839 8477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:51.891975 8477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:51.892269 8477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:51.892629 8477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:51.892691 8477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:51.892731 8477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:51.892760 8477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:51.898799 8477 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:38853
I20260621 05:20:51.898882 8605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:38853 every 8 connection(s)
I20260621 05:20:51.899266 8477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:20:51.899951 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 8477
I20260621 05:20:51.900091 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:20:51.900342 2176 external_mini_cluster.cc:1428] Setting key 027c0eb992eda87e21e0867956815239
WARNING: no policy specified for oryx/127.2.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.32.3@KRBTEST.COM" created.
Jun 21 05:20:51 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019251, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM
I20260621 05:20:51.917081 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:51.901251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:54455 (local address 127.2.32.62:45421)
0621 05:20:51.901309 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:51.901311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:51.901626 (+ 315us) server_negotiation.cc:407] Connection header received
0621 05:20:51.902169 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:51.902170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:51.902182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:51.902232 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:51.903334 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:51.903834 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:51.904940 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:51.905152 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:51.905224 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:51.913104 (+ 7880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:51.913107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:51.913107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:51.913119 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:51.914980 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:51.915654 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:51.915656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:51.915656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:51.915901 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:51.916300 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:51.916301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:51.916301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:51.916437 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:51.916497 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:51.916926 (+ 429us) server_negotiation.cc:299] Negotiation successful
0621 05:20:51.917011 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260621 05:20:51.917363 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:51.901235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:54455)
0621 05:20:51.901504 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:51.901545 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:51.902059 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:51.902300 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:51.902304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:51.902974 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:51.903242 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:51.903246 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:51.903927 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:51.903928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:51.904844 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:51.904846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:51.905241 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:51.905242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:51.905358 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:51.910768 (+ 5410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:51.910785 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:51.912951 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:51.915104 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:51.915113 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:51.915116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:51.915532 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:51.915993 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:51.915994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:51.915995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:51.916150 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:51.916514 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:51.916520 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:51.916721 (+ 201us) client_negotiation.cc:769] Sending connection context
0621 05:20:51.916848 (+ 127us) client_negotiation.cc:240] Negotiation successful
0621 05:20:51.917060 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":111,"threads_started":1}
I20260621 05:20:51.917848 8606 heartbeater.cc:347] Connected to a master server at 127.2.32.62:45421
I20260621 05:20:51.917945 8606 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:51.918172 8606 heartbeater.cc:511] Master 127.2.32.62:45421 requested a full tablet report, sending...
I20260621 05:20:51.918777 8281 ts_manager.cc:194] Registered new tserver with Master: 533996b4b53d4acb81ac269071c0adc5 (127.2.32.2:38853)
I20260621 05:20:51.919298 8281 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:54455
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.3.keytab.
Entry for principal oryx/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.3.keytab.
I20260621 05:20:51.931339 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:45421
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.3.keytab
--principal=oryx/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:52.018422 8611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:52.018712 8611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:52.018754 8611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:52.020090 8611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:52.020165 8611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:52.020210 8611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:20:52.021700 8611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.3.keytab
--principal=oryx/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:45421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:52.021958 8611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:52.022185 8611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:52.028030 8616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:52.028003 8619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:52.027999 8617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:52.028548 8611 server_base.cc:1034] running on GCE node
I20260621 05:20:52.028776 8611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:52.029057 8611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:52.030285 8611 hybrid_clock.cc:648] HybridClock initialized: now 1782019252030261 us; error 36 us; skew 500 ppm
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019252, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:52.033579 8611 init.cc:377] Logged in from keytab as oryx/127.2.32.3@KRBTEST.COM (short username oryx)
I20260621 05:20:52.034144 8611 webserver.cc:460] Webserver started at http://127.2.32.3:33955/ using document root <none> and password file <none>
I20260621 05:20:52.034441 8611 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:52.034518 8611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:52.034683 8611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:52.035971 8611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "73614dde1bc14e4889b823064cba9b95"
format_stamp: "Formatted at 2026-06-21 05:20:52 on dist-test-slave-b357"
server_key: "d1a56a1c1b48aa37a806f172f1be1da1"
server_key_iv: "048051f7bf434851338f667e6c034b80"
server_key_version: "encryptionkey@0"
I20260621 05:20:52.036369 8611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "73614dde1bc14e4889b823064cba9b95"
format_stamp: "Formatted at 2026-06-21 05:20:52 on dist-test-slave-b357"
server_key: "d1a56a1c1b48aa37a806f172f1be1da1"
server_key_iv: "048051f7bf434851338f667e6c034b80"
server_key_version: "encryptionkey@0"
I20260621 05:20:52.037804 8611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:20:52.038676 8626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:52.038937 8611 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:52.039047 8611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "73614dde1bc14e4889b823064cba9b95"
format_stamp: "Formatted at 2026-06-21 05:20:52 on dist-test-slave-b357"
server_key: "d1a56a1c1b48aa37a806f172f1be1da1"
server_key_iv: "048051f7bf434851338f667e6c034b80"
server_key_version: "encryptionkey@0"
I20260621 05:20:52.039158 8611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:52.069186 8611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:52.073016 8611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:52.073237 8611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:52.073604 8611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:52.073978 8611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:52.074038 8611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:52.074189 8611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:52.074250 8611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:52.082758 8611 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:38567
I20260621 05:20:52.082829 8739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:38567 every 8 connection(s)
I20260621 05:20:52.083300 8611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:20:52.087939 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 8611
I20260621 05:20:52.088043 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:20:52.088243 2176 external_mini_cluster.cc:1428] Setting key fb8f40363162801d822cdb58db94378b
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019252, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM
I20260621 05:20:52.096004 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.084090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:43913 (local address 127.2.32.62:45421)
0621 05:20:52.084173 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.084175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.084414 (+ 239us) server_negotiation.cc:407] Connection header received
0621 05:20:52.085051 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.085052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.085076 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.085146 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.086181 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.086859 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.088554 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.088841 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.088891 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.092316 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.092318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.092318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.092328 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.094488 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.094963 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.094964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.094965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.095036 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.095348 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.095349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.095349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.095470 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.095552 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.095830 (+ 278us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.095916 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 05:20:52.096213 8742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.084042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:43913)
0621 05:20:52.084267 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.084317 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.084928 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.085214 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.085218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.085827 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.086022 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.086054 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.087024 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.087025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.088437 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.088440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.088990 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.088991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.089124 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.089877 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.089897 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.092175 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.094594 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.094602 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.094605 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.094873 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.095120 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.095121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.095121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.095292 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.095543 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.095548 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.095691 (+ 143us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.095796 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.095958 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
I20260621 05:20:52.096751 8740 heartbeater.cc:347] Connected to a master server at 127.2.32.62:45421
I20260621 05:20:52.096871 8740 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:52.097091 8740 heartbeater.cc:511] Master 127.2.32.62:45421 requested a full tablet report, sending...
I20260621 05:20:52.097625 8281 ts_manager.cc:194] Registered new tserver with Master: 73614dde1bc14e4889b823064cba9b95 (127.2.32.3:38567)
I20260621 05:20:52.098038 8281 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:43913
I20260621 05:20:52.100560 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.32.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.32.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.32.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.61.keytab.
Entry for principal oryx/127.2.32.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.61.keytab.
I20260621 05:20:52.130332 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.61:38731
--webserver_interface=127.2.32.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.61.keytab
--principal=oryx/127.2.32.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:45421,127.2.32.61:38731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:52.212723 8745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:52.212939 8745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:52.212962 8745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:52.214069 8745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:52.214123 8745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:52.214140 8745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:52.214154 8745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:52.214169 8745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:52.215735 8745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:45421,127.2.32.61:38731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.61.keytab
--principal=oryx/127.2.32.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.61:38731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.32.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
W20260621 05:20:52.215838 8745 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.32.62:45421,127.2.32.61:38731'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260621 05:20:52.215972 8745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:52.216190 8745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:52.221086 8751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:52.221048 8753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:52.221289 8750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:52.221448 8745 server_base.cc:1034] running on GCE node
I20260621 05:20:52.221738 8745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:52.222002 8745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:52.223253 8745 hybrid_clock.cc:648] HybridClock initialized: now 1782019252223224 us; error 29 us; skew 500 ppm
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019252, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:52.226275 8745 init.cc:377] Logged in from keytab as oryx/127.2.32.61@KRBTEST.COM (short username oryx)
I20260621 05:20:52.226766 8745 webserver.cc:460] Webserver started at http://127.2.32.61:39005/ using document root <none> and password file <none>
I20260621 05:20:52.226994 8745 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:52.227061 8745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:52.227241 8745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:52.228276 8745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/data/instance:
uuid: "f0198561afcd4b85a28fa23b00c61a95"
format_stamp: "Formatted at 2026-06-21 05:20:52 on dist-test-slave-b357"
server_key: "763e1e9927a34ecfb8c85791e597a3f5"
server_key_iv: "7a4bf9ed2bb8a6c3c23a69a5b5db9a0c"
server_key_version: "encryptionkey@0"
I20260621 05:20:52.228642 8745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/wal/instance:
uuid: "f0198561afcd4b85a28fa23b00c61a95"
format_stamp: "Formatted at 2026-06-21 05:20:52 on dist-test-slave-b357"
server_key: "763e1e9927a34ecfb8c85791e597a3f5"
server_key_iv: "7a4bf9ed2bb8a6c3c23a69a5b5db9a0c"
server_key_version: "encryptionkey@0"
I20260621 05:20:52.230086 8745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:20:52.231317 8760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:52.231611 8745 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:52.231739 8745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/wal
uuid: "f0198561afcd4b85a28fa23b00c61a95"
format_stamp: "Formatted at 2026-06-21 05:20:52 on dist-test-slave-b357"
server_key: "763e1e9927a34ecfb8c85791e597a3f5"
server_key_iv: "7a4bf9ed2bb8a6c3c23a69a5b5db9a0c"
server_key_version: "encryptionkey@0"
I20260621 05:20:52.231837 8745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:52.252224 8745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:52.254711 8745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:52.254914 8745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:52.259570 8745 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.61:38731
I20260621 05:20:52.259644 8812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.61:38731 every 8 connection(s)
I20260621 05:20:52.260039 8745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/data/info.pb
I20260621 05:20:52.260972 8813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:52.262410 8813 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:20:52.266076 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 8745
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019252, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.61@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.32.61@KRBTEST.COM
I20260621 05:20:52.274000 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.263107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47994 (local address 127.2.32.62:45421)
0621 05:20:52.263197 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.263199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.263423 (+ 224us) server_negotiation.cc:407] Connection header received
0621 05:20:52.263922 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.263924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.263940 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.264030 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.264935 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.265575 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.266990 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.267303 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.267402 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.269874 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.269875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.269876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.269882 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.271961 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.272666 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.272667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.272668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.272757 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.273185 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.273187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.273187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.273373 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.273452 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.273762 (+ 310us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.273854 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 05:20:52.274235 8815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.263073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.0.0.1:47994)
0621 05:20:52.263277 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.263314 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.263773 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.264204 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.264210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.264547 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.264811 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.264816 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.265707 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.265709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.266869 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.266872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.267409 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.267410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.267535 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.268112 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.268120 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.269760 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.272108 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.272115 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.272116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.272503 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.272838 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.272839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.272840 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.273028 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.273452 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.273459 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.273618 (+ 159us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.273730 (+ 112us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.273964 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":122,"threads_started":1}
I20260621 05:20:52.274890 8813 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:20:52.277411 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.266695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44634 (local address 127.2.32.61:38731)
0621 05:20:52.266908 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.266914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.267083 (+ 169us) server_negotiation.cc:407] Connection header received
0621 05:20:52.267216 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.267219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.267251 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.267341 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.268025 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.268832 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.269752 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.270170 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.270248 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.273496 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.273504 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.273504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.273512 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.275915 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.276328 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.276331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.276331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.276401 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.276693 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.276694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.276695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.276886 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.277056 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.277101 (+ 45us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.277203 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":145,"threads_started":1}
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019252, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.61@KRBTEST.COM for oryx/127.2.32.61@KRBTEST.COM
I20260621 05:20:52.284003 8815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.275239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:38731 (local address 127.0.0.1:44650)
0621 05:20:52.275285 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.275297 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.275343 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.275614 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.275616 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.275852 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.275918 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.275919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.276884 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.276885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.278002 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.278004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.278486 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.278487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.278551 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.278939 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.278943 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.280443 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.282665 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.282666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.282667 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.282831 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.283207 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.283208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.283208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.283298 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.283720 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.283721 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.283759 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.283832 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.283928 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260621 05:20:52.284170 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.275284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44650 (local address 127.2.32.61:38731)
0621 05:20:52.275456 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.275458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.275469 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:20:52.275499 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.275499 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.275511 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.275583 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.276295 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.276796 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.278140 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.278372 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.278442 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.280566 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.280568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.280568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.280574 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.282570 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.282946 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.282947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.282948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.283029 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.283422 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.283423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.283424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.283623 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.283697 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.283912 (+ 215us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.284000 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":118,"threads_started":1}
I20260621 05:20:52.285836 8813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95: Bootstrap starting.
I20260621 05:20:52.286584 8813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:52.286898 8813 log.cc:824] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:52.287889 8813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95: No bootstrap required, opened a new log
I20260621 05:20:52.289382 8813 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } }
I20260621 05:20:52.289539 8813 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:52.289608 8813 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0198561afcd4b85a28fa23b00c61a95, State: Initialized, Role: FOLLOWER
I20260621 05:20:52.289783 8813 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } }
I20260621 05:20:52.290122 8813 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:52.290138 8820 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } } }
I20260621 05:20:52.290277 8820 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [sys.catalog]: This master's current role is: FOLLOWER
W20260621 05:20:52.292033 8831 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:20:52.292089 8831 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:20:52.293962 8745 master_runner.cc:428] Detected that this master f0198561afcd4b85a28fa23b00c61a95 is joining an existing cluster
I20260621 05:20:52.294039 8745 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.32.61:38731
I20260621 05:20:52.302048 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.295174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48002 (local address 127.2.32.62:45421)
0621 05:20:52.295216 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.295218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.295342 (+ 124us) server_negotiation.cc:407] Connection header received
0621 05:20:52.295430 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.295431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.295441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.295491 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.295859 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.296348 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.297520 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.297710 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.297788 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.299088 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.299089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.299089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.299096 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.300787 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.301208 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.301210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.301210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.301269 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.301454 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.301456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.301456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.301599 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.301662 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.301874 (+ 212us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.301956 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:20:52.302064 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.295169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.0.0.1:48002)
0621 05:20:52.295284 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.295295 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.295364 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.295557 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.295557 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.295733 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.295783 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.295784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.296431 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.296432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.297405 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.297407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.297810 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.297811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.297880 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.298529 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.298534 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.298982 (+ 448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.300882 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.300883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.300884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.301135 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.301322 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.301323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.301324 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.301387 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.301652 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.301652 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.301694 (+ 42us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.301862 (+ 168us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.301952 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":29,"thread_start_us":72,"threads_started":1}
I20260621 05:20:52.302419 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.294903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44664 (local address 127.2.32.61:38731)
0621 05:20:52.294965 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.294967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.295075 (+ 108us) server_negotiation.cc:407] Connection header received
0621 05:20:52.295201 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.295202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.295213 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.295242 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.295690 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.296454 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.297495 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.297800 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.297878 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.299071 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.299072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.299073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.299080 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.301194 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.301628 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.301629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.301629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.301690 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.301890 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.301891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.301891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.302000 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.302080 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.302253 (+ 173us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.302339 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260621 05:20:52.302754 8837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.294857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:38731 (local address 127.0.0.1:44664)
0621 05:20:52.295005 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.295014 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.295089 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.295302 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.295302 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.295508 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.295604 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.295606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.296524 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.296525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.297405 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.297407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.297878 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.297879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.297931 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.298387 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.298393 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.298982 (+ 589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.301270 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.301271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.301271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.301535 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.301773 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.301774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.301774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.301826 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.302120 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.302121 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.302158 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.302573 (+ 415us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.302660 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":48,"thread_start_us":105,"threads_started":1}
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019251, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.62@KRBTEST.COM for oryx/127.2.32.61@KRBTEST.COM
I20260621 05:20:52.312469 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.304662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:38731 (local address 127.0.0.1:44676)
0621 05:20:52.304781 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.304817 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.304910 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.305048 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.305051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.305310 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.305353 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.305354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.305947 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.305948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.306521 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.306523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.306876 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.306877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.306970 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.307419 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.307424 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.308904 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.310840 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.310855 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.310858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.311318 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.311642 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.311643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.311644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.311717 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.312056 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.312064 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.312130 (+ 66us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.312230 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.312368 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
I20260621 05:20:52.312476 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.304732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44676 (local address 127.2.32.61:38731)
0621 05:20:52.304771 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.304773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.304886 (+ 113us) server_negotiation.cc:407] Connection header received
0621 05:20:52.304978 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.304979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.304988 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.305043 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.305441 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.305871 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.306616 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.306806 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.306877 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.309001 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.309002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.309003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.309009 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.310691 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.311425 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.311427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.311428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.311513 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.311791 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.311792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.311793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.311951 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.312040 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.312273 (+ 233us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.312366 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:20:52.313544 8281 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.32.61:38731: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95"
member_type: NON_VOTER
last_known_addr {
host: "127.2.32.61"
port: 38731
}
attrs {
promote: true
}
}
dest_uuid: "087a170fca294b98b94961bab595c623"
cas_config_opid_index: -1
I20260621 05:20:52.313980 8281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: NON_VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: true } }
I20260621 05:20:52.314685 8841 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f0198561afcd4b85a28fa23b00c61a95 (127.2.32.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: NON_VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: true } } }
I20260621 05:20:52.315048 8787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:52.316120 8787 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 FOLLOWER]: Refusing update from remote peer 087a170fca294b98b94961bab595c623: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260621 05:20:52.316450 8840 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: NON_VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:52.317226 8841 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.32.61:38731
I20260621 05:20:52.317313 8841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "087a170fca294b98b94961bab595c623" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: NON_VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: true } } }
I20260621 05:20:52.317368 8841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:52.317441 8745 master.cc:537] Master@127.2.32.61:38731 shutting down...
I20260621 05:20:52.318527 8787 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f0198561afcd4b85a28fa23b00c61a95 (127.2.32.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: NON_VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: true } } }
I20260621 05:20:52.318822 8843 mvcc.cc:205] Tried to move back new op lower bound from 7299150854382923776 to 7299150854283554816. Current Snapshot: MvccSnapshot[applied={T|T < 7299150854293118976}]
I20260621 05:20:52.319695 8820 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 087a170fca294b98b94961bab595c623. Latest consensus state: current_term: 1 leader_uuid: "087a170fca294b98b94961bab595c623" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: NON_VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: true } } }
I20260621 05:20:52.319766 8820 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [sys.catalog]: This master's current role is: LEARNER
I20260621 05:20:52.319813 8820 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "087a170fca294b98b94961bab595c623" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: NON_VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: true } } }
I20260621 05:20:52.319864 8820 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [sys.catalog]: This master's current role is: LEARNER
I20260621 05:20:52.319893 8820 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "087a170fca294b98b94961bab595c623" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: NON_VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: true } } }
I20260621 05:20:52.319944 8820 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [sys.catalog]: This master's current role is: LEARNER
I20260621 05:20:52.321322 8745 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 LEARNER]: Raft consensus shutting down.
I20260621 05:20:52.321455 8745 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 LEARNER]: Raft consensus is shut down!
I20260621 05:20:52.321516 8745 tablet_replica.cc:330] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95: stopping tablet replica
I20260621 05:20:52.334555 8745 master.cc:552] Master@127.2.32.61:38731 shutdown complete.
I20260621 05:20:52.334648 8745 master_runner.cc:305] Clearing existing system tablet
I20260621 05:20:52.334824 8745 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:52.334875 8745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:52.335547 8745 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:20:52.336071 8851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:52.336239 8745 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:52.336316 8745 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:20:52.336347 8745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/wal
uuid: "f0198561afcd4b85a28fa23b00c61a95"
format_stamp: "Formatted at 2026-06-21 05:20:52 on dist-test-slave-b357"
server_key: "763e1e9927a34ecfb8c85791e597a3f5"
server_key_iv: "7a4bf9ed2bb8a6c3c23a69a5b5db9a0c"
server_key_version: "encryptionkey@0"
I20260621 05:20:52.337062 8745 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:52.338601 8745 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260621 05:20:52.338681 8745 log.cc:1196] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260621 05:20:52.338935 8745 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95: Deleting consensus metadata
I20260621 05:20:52.339067 8745 master_runner.cc:315] Copying system tablet from 127.2.32.62:45421
I20260621 05:20:52.339709 8745 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95: tablet copy: Beginning tablet copy session from remote peer at address 127.2.32.62:45421
I20260621 05:20:52.347182 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.340082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48016 (local address 127.2.32.62:45421)
0621 05:20:52.340132 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.340134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.340220 (+ 86us) server_negotiation.cc:407] Connection header received
0621 05:20:52.340264 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.340265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.340274 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.340323 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.340679 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.341168 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.342243 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.342458 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.342533 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.344009 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.344012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.344012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.344022 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.345857 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.346364 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.346366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.346366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.346427 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.346645 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.346646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.346646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.346757 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.346836 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.346987 (+ 151us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.347070 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260621 05:20:52.347204 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.339981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.0.0.1:48016)
0621 05:20:52.340115 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.340128 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.340203 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.340332 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.340333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.340537 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.340601 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.340603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.341244 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.341245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.342114 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.342118 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.342534 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.342535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.342612 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.343423 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.343431 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.343859 (+ 428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.345991 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.345992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.345993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.346280 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.346502 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.346503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.346503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.346576 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.346827 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.346828 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.346873 (+ 45us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.346962 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.347074 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":88,"threads_started":1}
I20260621 05:20:52.347609 8301 tablet_copy_service.cc:140] P 087a170fca294b98b94961bab595c623: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f0198561afcd4b85a28fa23b00c61a95 ({username='oryx', principal='oryx/127.2.32.61@KRBTEST.COM'} at 127.0.0.1:48016)
I20260621 05:20:52.347709 8301 tablet_copy_service.cc:161] P 087a170fca294b98b94961bab595c623: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f0198561afcd4b85a28fa23b00c61a95 at {username='oryx', principal='oryx/127.2.32.61@KRBTEST.COM'} at 127.0.0.1:48016: session id = f0198561afcd4b85a28fa23b00c61a95-00000000000000000000000000000000
I20260621 05:20:52.348326 8301 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 05:20:52.349080 8745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260621 05:20:52.350682 8745 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95: tablet copy: Starting download of 0 data blocks...
I20260621 05:20:52.350855 8745 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95: tablet copy: Starting download of 1 WAL segments...
I20260621 05:20:52.352036 8745 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 05:20:52.353256 8301 tablet_copy_service.cc:342] P 087a170fca294b98b94961bab595c623: Request end of tablet copy session f0198561afcd4b85a28fa23b00c61a95-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.32.61@KRBTEST.COM'} at 127.0.0.1:48016
I20260621 05:20:52.353338 8301 tablet_copy_service.cc:433] P 087a170fca294b98b94961bab595c623: ending tablet copy session f0198561afcd4b85a28fa23b00c61a95-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f0198561afcd4b85a28fa23b00c61a95
I20260621 05:20:52.356168 8745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:52.357710 8858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:52.357832 8859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:52.359965 8861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:52.360107 8745 server_base.cc:1034] running on GCE node
I20260621 05:20:52.360193 8745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:52.360357 8745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:52.361506 8745 hybrid_clock.cc:648] HybridClock initialized: now 1782019252361494 us; error 26 us; skew 500 ppm
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019252, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:52.363080 8745 init.cc:377] Logged in from keytab as oryx/127.2.32.61@KRBTEST.COM (short username oryx)
I20260621 05:20:52.363408 8745 webserver.cc:460] Webserver started at http://127.2.32.61:41343/ using document root <none> and password file <none>
I20260621 05:20:52.363534 8745 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:52.363590 8745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:52.364352 8745 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:52.365125 8868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:52.365335 8745 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:52.365397 8745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/wal
uuid: "f0198561afcd4b85a28fa23b00c61a95"
format_stamp: "Formatted at 2026-06-21 05:20:52 on dist-test-slave-b357"
server_key: "763e1e9927a34ecfb8c85791e597a3f5"
server_key_iv: "7a4bf9ed2bb8a6c3c23a69a5b5db9a0c"
server_key_version: "encryptionkey@0"
I20260621 05:20:52.365455 8745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:52.381440 8745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:52.383482 8745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:52.387120 8745 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.61:38731
I20260621 05:20:52.387247 8920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.61:38731 every 8 connection(s)
I20260621 05:20:52.387535 8745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-1/data/info.pb
I20260621 05:20:52.388576 8921 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:20:52.389166 8921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95: Bootstrap starting.
I20260621 05:20:52.391686 8921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:20:52.392081 8921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95: Bootstrap complete.
I20260621 05:20:52.392292 8921 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: NON_VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: true } }
I20260621 05:20:52.392387 8921 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0198561afcd4b85a28fa23b00c61a95, State: Initialized, Role: LEARNER
I20260621 05:20:52.392428 8921 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: NON_VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: true } }
I20260621 05:20:52.392612 8921 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:52.392628 8924 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: NON_VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: true } } }
I20260621 05:20:52.392736 8924 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [sys.catalog]: This master's current role is: LEARNER
I20260621 05:20:52.394723 8935 catalog_manager.cc:1259] Loaded cluster ID: 0352e39b766e443fb799d4ba4fbb2187
I20260621 05:20:52.394867 8935 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95: loading cluster ID for follower catalog manager: success
I20260621 05:20:52.395865 8935 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95: acquiring CA information for follower catalog manager: success
I20260621 05:20:52.396143 8935 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:20:52.436750 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.429623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44714 (local address 127.2.32.61:38731)
0621 05:20:52.429847 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.429849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.429858 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:20:52.429882 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.429883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.429895 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.430130 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.430706 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.431515 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.432235 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.432438 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.432526 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.433782 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.433783 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.433783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.433790 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.435585 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.435958 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.435959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.435960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.436019 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.436226 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.436227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.436228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.436340 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.436450 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.436548 (+ 98us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.436630 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for oryx/127.2.32.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.32.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.32.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.60.keytab.
Entry for principal oryx/127.2.32.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.60.keytab.
I20260621 05:20:52.466393 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.60:37877
--webserver_interface=127.2.32.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.60.keytab
--principal=oryx/127.2.32.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:45421,127.2.32.61:38731,127.2.32.60:37877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:20:52.559188 8939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:52.559412 8939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:52.559442 8939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:52.560653 8939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:20:52.560700 8939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:52.560717 8939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:52.560731 8939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:20:52.560745 8939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:20:52.562265 8939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:45421,127.2.32.61:38731,127.2.32.60:37877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.60.keytab
--principal=oryx/127.2.32.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.60:37877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.32.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:52.562546 8939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:52.562813 8939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:52.569132 8944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:52.569244 8947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:52.569244 8939 server_base.cc:1034] running on GCE node
W20260621 05:20:52.569103 8945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:52.569825 8939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:52.570273 8939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:52.571463 8939 hybrid_clock.cc:648] HybridClock initialized: now 1782019252571432 us; error 69 us; skew 500 ppm
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019252, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:52.574760 8939 init.cc:377] Logged in from keytab as oryx/127.2.32.60@KRBTEST.COM (short username oryx)
I20260621 05:20:52.575342 8939 webserver.cc:460] Webserver started at http://127.2.32.60:34125/ using document root <none> and password file <none>
I20260621 05:20:52.575589 8939 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:52.575657 8939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:52.575799 8939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:20:52.576853 8939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/data/instance:
uuid: "3440bc55a78b48ddb9f87e38c8bcf22a"
format_stamp: "Formatted at 2026-06-21 05:20:52 on dist-test-slave-b357"
server_key: "0bf8a2da5173e6cfe8679bd1229e390f"
server_key_iv: "7dfbaf4acbbf09090fe56c47cc9e13dc"
server_key_version: "encryptionkey@0"
I20260621 05:20:52.577204 8939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/wal/instance:
uuid: "3440bc55a78b48ddb9f87e38c8bcf22a"
format_stamp: "Formatted at 2026-06-21 05:20:52 on dist-test-slave-b357"
server_key: "0bf8a2da5173e6cfe8679bd1229e390f"
server_key_iv: "7dfbaf4acbbf09090fe56c47cc9e13dc"
server_key_version: "encryptionkey@0"
I20260621 05:20:52.578580 8939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:20:52.579532 8954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:52.579735 8939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:52.579854 8939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/wal
uuid: "3440bc55a78b48ddb9f87e38c8bcf22a"
format_stamp: "Formatted at 2026-06-21 05:20:52 on dist-test-slave-b357"
server_key: "0bf8a2da5173e6cfe8679bd1229e390f"
server_key_iv: "7dfbaf4acbbf09090fe56c47cc9e13dc"
server_key_version: "encryptionkey@0"
I20260621 05:20:52.579929 8939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:52.599563 8939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:52.602624 8939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:52.602869 8939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:52.607786 8939 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.60:37877
I20260621 05:20:52.607853 9006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.60:37877 every 8 connection(s)
I20260621 05:20:52.608275 8939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/data/info.pb
I20260621 05:20:52.609302 9007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:52.610891 9007 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:20:52.612617 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 8939
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019252, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.60@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.32.60@KRBTEST.COM
I20260621 05:20:52.621264 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.611800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48032 (local address 127.2.32.62:45421)
0621 05:20:52.611882 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.611884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.612083 (+ 199us) server_negotiation.cc:407] Connection header received
0621 05:20:52.612536 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.612538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.612564 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.612653 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.613457 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.613990 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.614942 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.615203 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.615264 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.617923 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.617925 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.617925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.617942 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.619723 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.620236 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.620237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.620238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.620309 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.620564 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.620565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.620565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.620704 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.620781 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.621075 (+ 294us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.621155 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 05:20:52.621835 9009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.611708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.0.0.1:48032)
0621 05:20:52.611927 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.611968 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.612422 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.612823 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.612826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.613173 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.613362 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.613366 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.614090 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.614090 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.614853 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.614855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.615287 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.615288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.615425 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.615963 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.615971 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.617785 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.619851 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.619858 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.619861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.620130 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.620384 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.620385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.620386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.620494 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.620796 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.620802 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.620919 (+ 117us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.621028 (+ 109us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.621211 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":16,"thread_start_us":143,"threads_started":1}
I20260621 05:20:52.622510 9007 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:20:52.623060 9010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.613184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.2.32.60:37877)
0621 05:20:52.613340 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.613344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.613361 (+ 17us) server_negotiation.cc:407] Connection header received
0621 05:20:52.613381 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.613383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.613423 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.613502 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.614072 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.614880 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.615816 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.616159 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.616215 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.619180 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.619189 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.619189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.619198 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.621227 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.621822 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.621824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.621825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.621972 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.622236 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.622238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.622238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.622454 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.622675 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.622845 (+ 170us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.622938 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":128,"thread_start_us":104,"threads_started":1}
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019252, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.60@KRBTEST.COM for oryx/127.2.32.61@KRBTEST.COM
I20260621 05:20:52.629242 9009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.622792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:38731 (local address 127.0.0.1:44716)
0621 05:20:52.623034 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.623045 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.623104 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.623276 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.623277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.623420 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.623463 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.623465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.624134 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.624135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.624624 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.624625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.624932 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.624932 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.624980 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.625376 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.625379 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.626471 (+ 1092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.628076 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.628078 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.628079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.628294 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.628541 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.628542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.628543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.628622 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.628948 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.628948 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.628994 (+ 46us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.629066 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.629161 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227}
I20260621 05:20:52.629276 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.622845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.2.32.61:38731)
0621 05:20:52.622884 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.622886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.623107 (+ 221us) server_negotiation.cc:407] Connection header received
0621 05:20:52.623208 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.623209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.623223 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.623273 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.623539 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.624072 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.624690 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.624871 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.624921 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.626572 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.626574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.626574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.626580 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.627948 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.628392 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.628393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.628394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.628445 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.628730 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.628732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.628732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.628874 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.628941 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.629120 (+ 179us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.629204 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:20:52.630146 9007 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.60" port: 37877 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019252, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.60@KRBTEST.COM for oryx/127.2.32.60@KRBTEST.COM
I20260621 05:20:52.637334 9009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.630480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:37877 (local address 127.0.0.1:35358)
0621 05:20:52.630533 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.630545 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.630637 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.630827 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.630828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.630968 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.631001 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.631003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.631725 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.631726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.632211 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.632213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.632592 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.632592 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.632642 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.633100 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.633105 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.634597 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.636314 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.636315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.636315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.636496 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.636666 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.636666 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.636667 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.636716 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.636969 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.636970 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.637032 (+ 62us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.637117 (+ 85us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.637224 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260621 05:20:52.637410 9010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.630661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.2.32.60:37877)
0621 05:20:52.630721 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.630723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.630737 (+ 14us) server_negotiation.cc:407] Connection header received
0621 05:20:52.630751 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.630752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.630759 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.630826 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.631093 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.631628 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.632293 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.632515 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.632594 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.634690 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.634691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.634692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.634698 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.636234 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.636562 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.636563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.636563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.636608 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.636766 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.636767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.636767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.636865 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.636916 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.637176 (+ 260us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.637273 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":40}
I20260621 05:20:52.639026 9007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a: Bootstrap starting.
I20260621 05:20:52.639940 9007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:52.640245 9007 log.cc:824] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:52.641026 9007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a: No bootstrap required, opened a new log
I20260621 05:20:52.642560 9007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 37877 } }
I20260621 05:20:52.642691 9007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:52.642714 9007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3440bc55a78b48ddb9f87e38c8bcf22a, State: Initialized, Role: FOLLOWER
I20260621 05:20:52.642838 9007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 37877 } }
I20260621 05:20:52.643229 9007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:52.643352 9012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 37877 } } }
I20260621 05:20:52.643424 9012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a [sys.catalog]: This master's current role is: FOLLOWER
W20260621 05:20:52.645020 9023 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:20:52.645080 9023 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:20:52.648778 8939 master_runner.cc:428] Detected that this master 3440bc55a78b48ddb9f87e38c8bcf22a is joining an existing cluster
I20260621 05:20:52.648876 8939 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.32.60:37877
I20260621 05:20:52.658185 9010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.650354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35366 (local address 127.2.32.60:37877)
0621 05:20:52.650392 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.650394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.650431 (+ 37us) server_negotiation.cc:407] Connection header received
0621 05:20:52.650533 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.650534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.650544 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.650576 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.651368 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.652152 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.653451 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.653755 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.653797 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.655037 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.655039 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.655039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.655045 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.656944 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.657387 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.657388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.657388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.657458 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.657699 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.657700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.657700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.657818 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.657884 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.658038 (+ 154us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.658123 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:20:52.658361 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.650145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:37877 (local address 127.0.0.1:35366)
0621 05:20:52.650352 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.650366 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.650460 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.650660 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.650661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.651055 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.651276 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.651279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.652249 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.652249 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.653352 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.653354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.653899 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.653900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.653957 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.654515 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.654519 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.654916 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.657042 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.657043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.657044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.657293 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.657530 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.657530 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.657531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.657612 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.657894 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.657895 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.657941 (+ 46us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.658021 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.658204 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":175,"thread_start_us":109,"threads_started":1}
I20260621 05:20:52.674513 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 8343
I20260621 05:20:52.680415 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:42265
--local_ip_for_outbound_sockets=127.2.32.1
--tserver_master_addrs=127.2.32.62:45421
--webserver_port=35091
--webserver_interface=127.2.32.1
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.1.keytab
--principal=oryx/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.32.62:45421,127.2.32.61:38731
--tserver_master_addrs=127.2.32.62:45421,127.2.32.61:38731,127.2.32.60:37877 with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:20:52.681581 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.650472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48040 (local address 127.2.32.62:45421)
0621 05:20:52.650528 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.650530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.650797 (+ 267us) server_negotiation.cc:407] Connection header received
0621 05:20:52.650894 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.650895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.650907 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.650937 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.651426 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.652192 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.653455 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.653755 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.653797 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.678480 (+ 24683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.678482 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.678483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.678492 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.680137 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.680650 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.680651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.680651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.680709 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.680945 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.680946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.680947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.681095 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.681142 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.681394 (+ 252us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.681483 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260621 05:20:52.681914 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.650582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.0.0.1:48040)
0621 05:20:52.650732 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.650743 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.650820 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.651004 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.651005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.651255 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.651335 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.651340 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.652287 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.652287 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.653351 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.653354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.653881 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.653881 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.653946 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.677976 (+ 24030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.677980 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.678348 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.680244 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.680245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.680246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.680553 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.680789 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.680790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.680790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.680863 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.681194 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.681195 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.681258 (+ 63us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.681679 (+ 421us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.681797 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":100,"thread_start_us":75,"threads_started":1}
I20260621 05:20:52.682132 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.650187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44724 (local address 127.2.32.61:38731)
0621 05:20:52.650243 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.650247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.650527 (+ 280us) server_negotiation.cc:407] Connection header received
0621 05:20:52.650629 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.650630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.650641 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.650672 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.651380 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.652141 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.653866 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.654150 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.654193 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.677957 (+ 23764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.677959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.677960 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.677970 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.680205 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.681017 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.681018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.681018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.681075 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.681266 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.681267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.681267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.681404 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.681444 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.681899 (+ 455us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.682009 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:20:52.682976 9029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.650159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:38731 (local address 127.0.0.1:44724)
0621 05:20:52.650439 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.650462 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.650556 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.650740 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.650740 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.651064 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.651281 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.651284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.652241 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.652241 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.653761 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.653764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.654272 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.654273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.654348 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.677359 (+ 23011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.677365 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.677826 (+ 461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.680306 (+ 2480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.680307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.680308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.680913 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.681144 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.681145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.681145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.681205 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.681490 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.681490 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.681532 (+ 42us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.682731 (+ 1199us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.682851 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":86,"thread_start_us":112,"threads_started":1}
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019251, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.62@KRBTEST.COM for oryx/127.2.32.60@KRBTEST.COM
I20260621 05:20:52.693971 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.684512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:37877 (local address 127.0.0.1:35378)
0621 05:20:52.684609 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.684621 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.684673 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.685002 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.685003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.685225 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.685298 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.685300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.686285 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.686286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.687171 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.687174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.687760 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.687762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.687838 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.688397 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.688403 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.690101 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.692348 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.692349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.692350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.692586 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.692989 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.692990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.692990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.693130 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.693512 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.693513 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.693580 (+ 67us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.693667 (+ 87us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.693849 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260621 05:20:52.693955 9010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.684748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35378 (local address 127.2.32.60:37877)
0621 05:20:52.684824 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.684827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.684839 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:20:52.684875 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.684876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.684889 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.684971 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.685387 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.686184 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.687290 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.687648 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.687760 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.690268 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.690270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.690271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.690280 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.692249 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.692689 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.692690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.692691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.692764 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.693271 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.693273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.693273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.693429 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.693507 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.693700 (+ 193us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.693848 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 05:20:52.695098 8281 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.32.60:37877: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a"
member_type: NON_VOTER
last_known_addr {
host: "127.2.32.60"
port: 37877
}
attrs {
promote: true
}
}
dest_uuid: "087a170fca294b98b94961bab595c623"
cas_config_opid_index: 5
I20260621 05:20:52.695562 8281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: NON_VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: true } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } }
I20260621 05:20:52.695935 8841 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3440bc55a78b48ddb9f87e38c8bcf22a (127.2.32.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: NON_VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: true } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } } }
I20260621 05:20:52.696372 8981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:52.696887 8841 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.32.60:37877
I20260621 05:20:52.696959 8841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "087a170fca294b98b94961bab595c623" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: NON_VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: true } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } } }
I20260621 05:20:52.697016 8841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:52.697294 8981 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a [term 1 FOLLOWER]: Refusing update from remote peer 087a170fca294b98b94961bab595c623: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260621 05:20:52.697358 8939 master.cc:537] Master@127.2.32.60:37877 shutting down...
I20260621 05:20:52.697880 8840 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 05:20:52.698422 8265 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260621 05:20:52.700402 8265 consensus_peers.cc:487] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 -> Peer 3440bc55a78b48ddb9f87e38c8bcf22a (127.2.32.60:37877): Couldn't send request to peer 3440bc55a78b48ddb9f87e38c8bcf22a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260621 05:20:52.700974 8939 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:20:52.701066 8939 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:52.701115 8939 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a: stopping tablet replica
I20260621 05:20:52.704010 8939 master.cc:552] Master@127.2.32.60:37877 shutdown complete.
I20260621 05:20:52.704056 8939 master_runner.cc:305] Clearing existing system tablet
I20260621 05:20:52.704181 8939 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:52.704222 8939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:52.704246 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.697146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44728 (local address 127.2.32.61:38731)
0621 05:20:52.698922 (+ 1776us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.698924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.698933 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:20:52.698957 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.698958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.699006 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.699081 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:52.699288 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.700021 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.703047 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.703649 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.703704 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.703766 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.704134 (+ 368us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.704177 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1753}
I20260621 05:20:52.704591 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.696042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:38731 (local address 127.0.0.1:44728)
0621 05:20:52.697818 (+ 1776us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.697831 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.697882 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.699110 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.699111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.699113 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:52.699188 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.699190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.700606 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.700608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.702937 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.702940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.703834 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.703835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.703901 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.703920 (+ 19us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.703976 (+ 56us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.704019 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1754}
I20260621 05:20:52.704936 8939 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:20:52.704980 8895 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 LEARNER]: Refusing update from remote peer 087a170fca294b98b94961bab595c623: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260621 05:20:52.705178 8840 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: NON_VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260621 05:20:52.705353 9035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:52.705443 8939 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:52.705477 8939 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:52.705497 8939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/wal
uuid: "3440bc55a78b48ddb9f87e38c8bcf22a"
format_stamp: "Formatted at 2026-06-21 05:20:52 on dist-test-slave-b357"
server_key: "0bf8a2da5173e6cfe8679bd1229e390f"
server_key_iv: "7dfbaf4acbbf09090fe56c47cc9e13dc"
server_key_version: "encryptionkey@0"
I20260621 05:20:52.705633 8895 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3440bc55a78b48ddb9f87e38c8bcf22a (127.2.32.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: NON_VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: true } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } } }
I20260621 05:20:52.706189 8939 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:52.707574 8939 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260621 05:20:52.707630 8939 log.cc:1196] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260621 05:20:52.707814 8939 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a: Deleting consensus metadata
I20260621 05:20:52.708019 8939 master_runner.cc:315] Copying system tablet from 127.2.32.62:45421
I20260621 05:20:52.707944 8924 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 087a170fca294b98b94961bab595c623. Latest consensus state: current_term: 1 leader_uuid: "087a170fca294b98b94961bab595c623" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: NON_VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: true } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } } }
I20260621 05:20:52.708057 8924 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [sys.catalog]: This master's current role is: LEARNER
I20260621 05:20:52.708102 8924 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "087a170fca294b98b94961bab595c623" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: NON_VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: true } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } } }
I20260621 05:20:52.708154 8924 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [sys.catalog]: This master's current role is: LEARNER
I20260621 05:20:52.708577 8939 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a: tablet copy: Beginning tablet copy session from remote peer at address 127.2.32.62:45421
I20260621 05:20:52.715755 8840 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623: attempting to promote NON_VOTER f0198561afcd4b85a28fa23b00c61a95 to VOTER
I20260621 05:20:52.715992 8840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: false } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } }
I20260621 05:20:52.716231 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.708909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48042 (local address 127.2.32.62:45421)
0621 05:20:52.708946 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.708948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.709069 (+ 121us) server_negotiation.cc:407] Connection header received
0621 05:20:52.709119 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.709120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.709135 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.709207 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.709551 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.710300 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.711336 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.711625 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.711665 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.712883 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.712885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.712886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.712898 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.714793 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.715454 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.715456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.715456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.715524 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.715712 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.715713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.715713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.715832 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.715874 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.716070 (+ 196us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.716152 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:20:52.716933 8895 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 LEARNER]: Refusing update from remote peer 087a170fca294b98b94961bab595c623: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260621 05:20:52.717028 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.708852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.0.0.1:48042)
0621 05:20:52.708997 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.709008 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.709064 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.709213 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.709213 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.709421 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.709478 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.709480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.710389 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.710390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.711237 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.711239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.711740 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.711741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.711802 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.712359 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.712364 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.712754 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.714984 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.714986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.714986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.715342 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.715594 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.715594 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.715595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.715649 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.715917 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.715918 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.715962 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.716332 (+ 370us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.716554 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
I20260621 05:20:52.717234 8301 tablet_copy_service.cc:140] P 087a170fca294b98b94961bab595c623: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3440bc55a78b48ddb9f87e38c8bcf22a ({username='oryx', principal='oryx/127.2.32.60@KRBTEST.COM'} at 127.0.0.1:48042)
I20260621 05:20:52.717283 8301 tablet_copy_service.cc:161] P 087a170fca294b98b94961bab595c623: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3440bc55a78b48ddb9f87e38c8bcf22a at {username='oryx', principal='oryx/127.2.32.60@KRBTEST.COM'} at 127.0.0.1:48042: session id = 3440bc55a78b48ddb9f87e38c8bcf22a-00000000000000000000000000000000
I20260621 05:20:52.717712 8841 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260621 05:20:52.717813 8301 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 05:20:52.718261 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.716620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:37877 (local address 127.0.0.1:35382)
0621 05:20:52.718189 (+ 1569us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.718235 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:37877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1553}
I20260621 05:20:52.718266 8939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260621 05:20:52.718653 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.718487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:37877 (local address 127.0.0.1:35384)
0621 05:20:52.718605 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.718631 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:37877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260621 05:20:52.718719 8265 consensus_peers.cc:487] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 -> Peer 3440bc55a78b48ddb9f87e38c8bcf22a (127.2.32.60:37877): Couldn't send request to peer 3440bc55a78b48ddb9f87e38c8bcf22a. Status: Network error: Client connection negotiation failed: client connection to 127.2.32.60:37877: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260621 05:20:52.719203 9044 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f0198561afcd4b85a28fa23b00c61a95 (127.2.32.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: false } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } } }
I20260621 05:20:52.719413 8939 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a: tablet copy: Starting download of 0 data blocks...
I20260621 05:20:52.719362 8895 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f0198561afcd4b85a28fa23b00c61a95 (127.2.32.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: false } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } } }
I20260621 05:20:52.719535 8939 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a: tablet copy: Starting download of 1 WAL segments...
I20260621 05:20:52.720667 9037 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "087a170fca294b98b94961bab595c623" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: false } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } } }
I20260621 05:20:52.720825 9037 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:20:52.721108 8939 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 05:20:52.722004 8301 tablet_copy_service.cc:342] P 087a170fca294b98b94961bab595c623: Request end of tablet copy session 3440bc55a78b48ddb9f87e38c8bcf22a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.32.60@KRBTEST.COM'} at 127.0.0.1:48042
I20260621 05:20:52.722059 8301 tablet_copy_service.cc:433] P 087a170fca294b98b94961bab595c623: ending tablet copy session 3440bc55a78b48ddb9f87e38c8bcf22a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3440bc55a78b48ddb9f87e38c8bcf22a
I20260621 05:20:52.724432 8841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "087a170fca294b98b94961bab595c623" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: false } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } } }
I20260621 05:20:52.724527 8841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:52.724443 9044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "087a170fca294b98b94961bab595c623" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: false } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } } }
I20260621 05:20:52.724695 9044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623 [sys.catalog]: This master's current role is: LEADER
I20260621 05:20:52.724948 8939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:52.726176 9045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:52.728140 9048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:52.728353 8939 server_base.cc:1034] running on GCE node
W20260621 05:20:52.734181 9046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:52.734349 8939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:52.734504 8939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:52.735651 8939 hybrid_clock.cc:648] HybridClock initialized: now 1782019252735623 us; error 45 us; skew 500 ppm
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019252, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:52.737182 8939 init.cc:377] Logged in from keytab as oryx/127.2.32.60@KRBTEST.COM (short username oryx)
I20260621 05:20:52.737426 8939 webserver.cc:460] Webserver started at http://127.2.32.60:43021/ using document root <none> and password file <none>
I20260621 05:20:52.737540 8939 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:52.737574 8939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:52.738286 8939 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:20:52.739004 9055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:52.739432 8939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:52.739486 8939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/wal
uuid: "3440bc55a78b48ddb9f87e38c8bcf22a"
format_stamp: "Formatted at 2026-06-21 05:20:52 on dist-test-slave-b357"
server_key: "0bf8a2da5173e6cfe8679bd1229e390f"
server_key_iv: "7dfbaf4acbbf09090fe56c47cc9e13dc"
server_key_version: "encryptionkey@0"
I20260621 05:20:52.739538 8939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:52.770983 8939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:52.772902 8939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:52.776422 8939 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.60:37877
I20260621 05:20:52.776527 9107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.60:37877 every 8 connection(s)
I20260621 05:20:52.776772 8939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/master-2/data/info.pb
I20260621 05:20:52.778110 9108 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:20:52.778616 9108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a: Bootstrap starting.
I20260621 05:20:52.781621 9108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260621 05:20:52.781868 9108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a: Bootstrap complete.
I20260621 05:20:52.782060 9108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: NON_VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: true } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } }
I20260621 05:20:52.782202 9108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3440bc55a78b48ddb9f87e38c8bcf22a, State: Initialized, Role: LEARNER
I20260621 05:20:52.782248 9108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: false } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } }
I20260621 05:20:52.782482 9108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:20:52.782471 9111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: NON_VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: true } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: false } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } } }
I20260621 05:20:52.782745 9111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a [sys.catalog]: This master's current role is: LEARNER
I20260621 05:20:52.784111 9122 catalog_manager.cc:1259] Loaded cluster ID: 0352e39b766e443fb799d4ba4fbb2187
I20260621 05:20:52.784212 9122 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a: loading cluster ID for follower catalog manager: success
I20260621 05:20:52.785534 9122 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a: acquiring CA information for follower catalog manager: success
I20260621 05:20:52.785859 9122 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3440bc55a78b48ddb9f87e38c8bcf22a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 05:20:52.792227 9032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:52.792474 9032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:52.792519 9032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:52.793995 9032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:52.794064 9032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:52.794109 9032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:20:52.795776 9032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.1.keytab
--principal=oryx/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:42265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=35091
--tserver_master_addrs=127.2.32.62:45421,127.2.32.61:38731,127.2.32.60:37877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:52.796011 9032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:52.796248 9032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:52.802711 9130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:52.802714 9128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:52.802731 9127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:52.803310 9032 server_base.cc:1034] running on GCE node
I20260621 05:20:52.803504 9032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:52.803771 9032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:52.804967 9032 hybrid_clock.cc:648] HybridClock initialized: now 1782019252804945 us; error 36 us; skew 500 ppm
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019252, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:52.807576 9032 init.cc:377] Logged in from keytab as oryx/127.2.32.1@KRBTEST.COM (short username oryx)
I20260621 05:20:52.808030 9032 webserver.cc:460] Webserver started at http://127.2.32.1:35091/ using document root <none> and password file <none>
I20260621 05:20:52.808230 9032 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:52.808277 9032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:52.809542 9032 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:52.810158 9137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:52.810307 9032 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:20:52.810384 9032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "4b7d4becc8c54574807da6bc9cddabf1"
format_stamp: "Formatted at 2026-06-21 05:20:51 on dist-test-slave-b357"
server_key: "b689da97e57cc44b0d144a783c000927"
server_key_iv: "cae37d898fd3ccc8075ed816a946f6f5"
server_key_version: "encryptionkey@0"
I20260621 05:20:52.810685 9032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:52.820078 9032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:52.821905 9032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:52.822227 9032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:52.822546 9032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:52.822970 9032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:52.823010 9032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:52.823036 9032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:52.823048 9032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:52.829272 9032 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:42265
I20260621 05:20:52.829420 9250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:42265 every 8 connection(s)
I20260621 05:20:52.829710 9032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019252, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019252, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.61@KRBTEST.COM
I20260621 05:20:52.837956 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 9032
I20260621 05:20:52.838140 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 8477
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019252, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.60@KRBTEST.COM
I20260621 05:20:52.841791 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.830694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:45775 (local address 127.2.32.62:45421)
0621 05:20:52.830800 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.830802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.830906 (+ 104us) server_negotiation.cc:407] Connection header received
0621 05:20:52.831798 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.831799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.831813 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.831851 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.832619 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.833125 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.834589 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.834901 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.834985 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.837731 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.837733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.837733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.837744 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.840070 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.840541 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.840542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.840543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.840607 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.840905 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.840907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.840907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.841068 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.841156 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.841564 (+ 408us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.841683 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 05:20:52.842160 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.830409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:45775)
0621 05:20:52.830844 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.830853 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.831672 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.831935 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.831936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.832281 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.832518 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.832523 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.833243 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.833244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.834500 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.834503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.835207 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.835208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.835361 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.835979 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.835985 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.837613 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.840160 (+ 2547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.840167 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.840169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.840454 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.840672 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.840672 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.840673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.840811 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.841199 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.841206 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.841361 (+ 155us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.841531 (+ 170us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.841867 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":49,"thread_start_us":107,"threads_started":1}
I20260621 05:20:52.843549 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.830410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:37877 (local address 127.2.32.1:52179)
0621 05:20:52.830665 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.830677 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.831672 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.831927 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.831927 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.832290 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.832518 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.832524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.834992 (+ 2468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.834993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.836218 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.836221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.836932 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.836934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.837024 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.837785 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.837791 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.839657 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.842107 (+ 2450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.842108 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.842109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.842368 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.842603 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.842604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.842604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.842685 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.843034 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.843035 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.843085 (+ 50us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.843256 (+ 171us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.843419 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":25,"thread_start_us":99,"threads_started":1}
I20260621 05:20:52.844120 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.830734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:52179 (local address 127.2.32.60:37877)
0621 05:20:52.830860 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.830862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.830871 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:20:52.831817 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.831819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.831832 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.831905 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.834106 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.834897 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.836363 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.836827 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.836929 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.839768 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.839769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.839770 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.839780 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.841992 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.842463 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.842464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.842464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.842529 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.842756 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.842757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.842757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.842917 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.842991 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.843854 (+ 863us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.843961 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":49,"threads_started":1}
I20260621 05:20:52.844509 9251 heartbeater.cc:347] Connected to a master server at 127.2.32.60:37877
I20260621 05:20:52.844606 9251 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:52.844794 9251 heartbeater.cc:511] Master 127.2.32.60:37877 requested a full tablet report, sending...
I20260621 05:20:52.845319 9064 ts_manager.cc:194] Registered new tserver with Master: 4b7d4becc8c54574807da6bc9cddabf1 (127.2.32.1:42265)
I20260621 05:20:52.845613 9252 heartbeater.cc:347] Connected to a master server at 127.2.32.62:45421
I20260621 05:20:52.845685 9252 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:52.845757 9252 heartbeater.cc:511] Master 127.2.32.62:45421 requested a full tablet report, sending...
I20260621 05:20:52.846068 8281 ts_manager.cc:194] Re-registered known tserver with Master: 4b7d4becc8c54574807da6bc9cddabf1 (127.2.32.1:42265)
I20260621 05:20:52.846099 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:38853
--local_ip_for_outbound_sockets=127.2.32.2
--tserver_master_addrs=127.2.32.62:45421
--webserver_port=37883
--webserver_interface=127.2.32.2
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.2.keytab
--principal=oryx/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.32.62:45421,127.2.32.61:38731
--tserver_master_addrs=127.2.32.62:45421,127.2.32.61:38731,127.2.32.60:37877 with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:20:52.846596 8281 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:45775
I20260621 05:20:52.847172 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.830751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:49859 (local address 127.2.32.61:38731)
0621 05:20:52.830810 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.830813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.830824 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:20:52.831751 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.831752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.831763 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.831820 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.833252 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.833997 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.834849 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.835177 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.835222 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.843933 (+ 8711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.843946 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.843946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.843954 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.845632 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.846037 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.846038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.846038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.846093 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.846489 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.846490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.846490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.846646 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.846713 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.846900 (+ 187us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.846991 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260621 05:20:52.847536 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.830410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:38731 (local address 127.2.32.1:49859)
0621 05:20:52.830585 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.830624 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.831641 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.831889 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.831892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.832281 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.832518 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.832524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.834069 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.834070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.834749 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.834751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.835349 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.835349 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.835399 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.835851 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.835863 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.843831 (+ 7968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.845707 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.845708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.845708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.845935 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.846334 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.846335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.846335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.846406 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.846743 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.846744 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.846790 (+ 46us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.847250 (+ 460us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.847386 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":6,"thread_start_us":98,"threads_started":1}
I20260621 05:20:52.847915 9253 heartbeater.cc:347] Connected to a master server at 127.2.32.61:38731
I20260621 05:20:52.847957 9253 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:52.848032 9253 heartbeater.cc:511] Master 127.2.32.61:38731 requested a full tablet report, sending...
I20260621 05:20:52.848414 8885 ts_manager.cc:194] Registered new tserver with Master: 4b7d4becc8c54574807da6bc9cddabf1 (127.2.32.1:42265)
W20260621 05:20:52.932492 9259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:52.932700 9259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:52.932726 9259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:52.933837 9259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:52.933888 9259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:52.933917 9259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:20:52.935670 9259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.2.keytab
--principal=oryx/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:38853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=37883
--tserver_master_addrs=127.2.32.62:45421,127.2.32.61:38731,127.2.32.60:37877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:52.935914 9259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:52.936183 9259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:52.942296 9265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:52.942230 9267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:52.942186 9264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:52.942966 9259 server_base.cc:1034] running on GCE node
I20260621 05:20:52.943357 9259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:52.943701 9259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:52.944896 9259 hybrid_clock.cc:648] HybridClock initialized: now 1782019252944887 us; error 60 us; skew 500 ppm
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019252, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:52.947647 9259 init.cc:377] Logged in from keytab as oryx/127.2.32.2@KRBTEST.COM (short username oryx)
I20260621 05:20:52.948407 9259 webserver.cc:460] Webserver started at http://127.2.32.2:37883/ using document root <none> and password file <none>
I20260621 05:20:52.948768 9259 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:52.948904 9259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:52.950397 9259 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:52.951287 9274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:52.951651 9259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:20:52.951817 9259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "533996b4b53d4acb81ac269071c0adc5"
format_stamp: "Formatted at 2026-06-21 05:20:51 on dist-test-slave-b357"
server_key: "28562493b8c782540bcaac537cab7813"
server_key_iv: "d73fda16091034022942a9b2f170044d"
server_key_version: "encryptionkey@0"
I20260621 05:20:52.952291 9259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:52.971511 9259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:52.973990 9259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:52.974161 9259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:52.974404 9259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:52.974702 9259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:52.974766 9259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:52.974807 9259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:52.974839 9259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:52.982203 9259 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:38853
I20260621 05:20:52.982287 9387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:38853 every 8 connection(s)
I20260621 05:20:52.982643 9259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019252, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.60@KRBTEST.COM
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019252, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM
Jun 21 05:20:52 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019252, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.61@KRBTEST.COM
I20260621 05:20:52.992064 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 9259
I20260621 05:20:52.992230 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 8611
I20260621 05:20:52.996178 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.983444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:55093 (local address 127.2.32.62:45421)
0621 05:20:52.983488 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.983490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.983881 (+ 391us) server_negotiation.cc:407] Connection header received
0621 05:20:52.984987 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.984988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.985002 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.985070 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.985787 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.986626 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.987714 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.988015 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.988081 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.990736 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.990738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.990738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.990744 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.993526 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.994996 (+ 1470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.994997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.994998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.995063 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.995315 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.995316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.995316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.995540 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.995598 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.995995 (+ 397us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.996082 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:20:52.996183 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.983377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:55797 (local address 127.2.32.61:38731)
0621 05:20:52.983428 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.983430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.983764 (+ 334us) server_negotiation.cc:407] Connection header received
0621 05:20:52.985026 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.985027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.985041 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.985093 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.985787 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.986529 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.987851 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.988071 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.988114 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.991332 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.991333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.991334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.991340 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.994166 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.994932 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.994933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.994934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.995005 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.995314 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.995315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.995315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.995540 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.995598 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.996031 (+ 433us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.996102 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260621 05:20:52.996786 9392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.983384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:55093)
0621 05:20:52.983820 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.983830 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.984855 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.985153 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.985157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.985537 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.985682 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.985688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.986709 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.986709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.987616 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.987618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.988108 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.988109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.988247 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.988930 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.988943 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.990630 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.994248 (+ 3618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.994253 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.994254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.994836 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.995117 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.995117 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.995118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.995226 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.995661 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.995669 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.995832 (+ 163us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.996307 (+ 475us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.996608 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":245,"thread_start_us":98,"threads_started":1}
I20260621 05:20:52.996786 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.983384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:38731 (local address 127.2.32.2:55797)
0621 05:20:52.983602 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.983672 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.984881 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.985153 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.985157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.985540 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.985682 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.985688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.986624 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.986625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.987725 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.987726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.988187 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.988187 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.988247 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.989056 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.989060 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.991219 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.994270 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.994273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.994273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.994819 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.995063 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.995064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.995064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.995226 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.995661 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.995669 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.995832 (+ 163us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.996307 (+ 475us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.996541 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":120,"thread_start_us":112,"threads_started":1}
I20260621 05:20:52.998358 9390 heartbeater.cc:347] Connected to a master server at 127.2.32.61:38731
I20260621 05:20:52.998459 9390 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:52.998636 9390 heartbeater.cc:511] Master 127.2.32.61:38731 requested a full tablet report, sending...
I20260621 05:20:52.998831 9389 heartbeater.cc:347] Connected to a master server at 127.2.32.62:45421
I20260621 05:20:52.998899 9389 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:52.998961 9389 heartbeater.cc:511] Master 127.2.32.62:45421 requested a full tablet report, sending...
I20260621 05:20:52.999073 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:38567
--local_ip_for_outbound_sockets=127.2.32.3
--tserver_master_addrs=127.2.32.62:45421
--webserver_port=33955
--webserver_interface=127.2.32.3
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.3.keytab
--principal=oryx/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.32.62:45421,127.2.32.61:38731
--tserver_master_addrs=127.2.32.62:45421,127.2.32.61:38731,127.2.32.60:37877 with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:20:52.999329 8885 ts_manager.cc:194] Registered new tserver with Master: 533996b4b53d4acb81ac269071c0adc5 (127.2.32.2:38853)
I20260621 05:20:52.999476 8281 ts_manager.cc:194] Re-registered known tserver with Master: 533996b4b53d4acb81ac269071c0adc5 (127.2.32.2:38853)
I20260621 05:20:52.999642 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.983717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:37877 (local address 127.2.32.2:38145)
0621 05:20:52.983830 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:52.983840 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:52.984855 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:52.985157 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:52.985158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:52.985485 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:52.985681 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.985686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.986551 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.986552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.987723 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:52.987725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.988219 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.988220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:52.988292 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.988715 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:52.988723 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:52.990163 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:52.995796 (+ 5633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.995797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.995798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.996048 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.998836 (+ 2788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:52.998837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:52.998837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:52.998931 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:52.999287 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:52.999288 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:52.999327 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 05:20:52.999409 (+ 82us) client_negotiation.cc:240] Negotiation successful
0621 05:20:52.999499 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":85,"thread_start_us":71,"threads_started":1}
I20260621 05:20:52.999645 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:52.983800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:38145 (local address 127.2.32.60:37877)
0621 05:20:52.983845 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:52.983847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:52.983883 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:20:52.985055 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:52.985057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:52.985067 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:52.985116 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:52.985778 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.986280 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.987806 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:52.988143 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:52.988205 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:52.990371 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:52.990373 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:52.990374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:52.990386 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:52.992503 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.996145 (+ 3642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.996146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.996146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.996216 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:52.999017 (+ 2801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:52.999018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:52.999019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:52.999184 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:52.999238 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:52.999463 (+ 225us) server_negotiation.cc:299] Negotiation successful
0621 05:20:52.999562 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:20:52.999903 8281 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:55093
I20260621 05:20:53.000125 9388 heartbeater.cc:347] Connected to a master server at 127.2.32.60:37877
I20260621 05:20:53.000178 9388 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:53.000253 9388 heartbeater.cc:511] Master 127.2.32.60:37877 requested a full tablet report, sending...
I20260621 05:20:53.000646 9064 ts_manager.cc:194] Registered new tserver with Master: 533996b4b53d4acb81ac269071c0adc5 (127.2.32.2:38853)
W20260621 05:20:53.084746 9395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:20:53.084967 9395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:20:53.085016 9395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:20:53.086156 9395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:20:53.086222 9395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:20:53.086268 9395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:20:53.087843 9395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.3.keytab
--principal=oryx/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:38567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=33955
--tserver_master_addrs=127.2.32.62:45421,127.2.32.61:38731,127.2.32.60:37877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:20:53.088117 9395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:20:53.088351 9395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:20:53.093721 9403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:53.093678 9401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:20:53.093829 9400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:20:53.094013 9395 server_base.cc:1034] running on GCE node
I20260621 05:20:53.094373 9395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:20:53.094643 9395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:20:53.095837 9395 hybrid_clock.cc:648] HybridClock initialized: now 1782019253095838 us; error 45 us; skew 500 ppm
Jun 21 05:20:53 dist-test-slave-b357 krb5kdc[8236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019253, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:20:53.098927 9395 init.cc:377] Logged in from keytab as oryx/127.2.32.3@KRBTEST.COM (short username oryx)
I20260621 05:20:53.099436 9395 webserver.cc:460] Webserver started at http://127.2.32.3:33955/ using document root <none> and password file <none>
I20260621 05:20:53.099691 9395 fs_manager.cc:359] Metadata directory not provided
I20260621 05:20:53.099870 9395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:20:53.101683 9395 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:20:53.102398 9410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:53.102608 9395 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:20:53.102716 9395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "73614dde1bc14e4889b823064cba9b95"
format_stamp: "Formatted at 2026-06-21 05:20:52 on dist-test-slave-b357"
server_key: "d1a56a1c1b48aa37a806f172f1be1da1"
server_key_iv: "048051f7bf434851338f667e6c034b80"
server_key_version: "encryptionkey@0"
I20260621 05:20:53.103062 9395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:20:53.125263 9395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:20:53.127524 9395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:20:53.127713 9395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:20:53.127964 9395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:20:53.128293 9395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:20:53.128364 9395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:53.128410 9395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:20:53.128443 9395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:20:53.135471 9395 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:38567
I20260621 05:20:53.135530 9523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:38567 every 8 connection(s)
I20260621 05:20:53.135869 9395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:20:53 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019253, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.61@KRBTEST.COM
Jun 21 05:20:53 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019253, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM
Jun 21 05:20:53 dist-test-slave-b357 krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019253, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.60@KRBTEST.COM
I20260621 05:20:53.145283 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 9395
I20260621 05:20:53.147310 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.136905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:42739 (local address 127.2.32.61:38731)
0621 05:20:53.136953 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:53.136955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:53.136966 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:20:53.137511 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:53.137512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:53.137526 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:53.137590 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:53.138615 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.139390 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.140748 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.140990 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.141078 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.143403 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:53.143405 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:53.143405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:53.143413 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:53.145321 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:53.145882 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:53.145884 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:53.145884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:53.145941 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:53.146242 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:53.146243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:53.146244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:53.146544 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:53.146607 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:53.147097 (+ 490us) server_negotiation.cc:299] Negotiation successful
0621 05:20:53.147198 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:20:53.147472 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.136735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:54693 (local address 127.2.32.60:37877)
0621 05:20:53.136781 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:53.136784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:53.137846 (+ 1062us) server_negotiation.cc:407] Connection header received
0621 05:20:53.137921 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:53.137922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:53.137933 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:53.137995 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:53.138601 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.139283 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.140783 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.141077 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.141121 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.144393 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:53.144394 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:53.144395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:53.144402 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:53.146264 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:53.146659 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:53.146661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:53.146661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:53.146714 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:53.146904 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:53.146905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:53.146906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:53.147027 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:53.147071 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:53.147298 (+ 227us) server_negotiation.cc:299] Negotiation successful
0621 05:20:53.147390 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:20:53.147809 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.136610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:38731 (local address 127.2.32.3:42739)
0621 05:20:53.136781 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:53.136824 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:53.137404 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:53.137675 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:53.137679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:53.138235 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:53.138506 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:53.138512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.139487 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.139487 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:53.140641 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:53.140643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.141088 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.141089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:53.141222 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.141726 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:53.141736 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:53.143315 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:53.145431 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:53.145437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:53.145439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:53.145792 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:53.146019 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:53.146020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:53.146020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:53.146161 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:53.146945 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:53.146946 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:53.146981 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 05:20:53.147413 (+ 432us) client_negotiation.cc:240] Negotiation successful
0621 05:20:53.147581 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":143,"thread_start_us":98,"threads_started":1}
I20260621 05:20:53.147809 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.136611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:37877 (local address 127.2.32.3:54693)
0621 05:20:53.137770 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:53.137778 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:53.137851 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:53.138002 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:53.138003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:53.138341 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:53.138505 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:53.138510 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.139389 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.139390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:53.140688 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:53.140690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.141204 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.141205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:53.141271 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.141967 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:53.141971 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:53.144283 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:53.146347 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:53.146347 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:53.146348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:53.146575 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:53.146781 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:53.146782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:53.146782 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:53.146833 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:53.147117 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:53.147117 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:53.147186 (+ 69us) client_negotiation.cc:769] Sending connection context
0621 05:20:53.147590 (+ 404us) client_negotiation.cc:240] Negotiation successful
0621 05:20:53.147666 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1138,"mutex_wait_us":426,"thread_start_us":72,"threads_started":1}
I20260621 05:20:53.148257 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.136603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:43273 (local address 127.2.32.62:45421)
0621 05:20:53.136658 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:53.136660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:53.137111 (+ 451us) server_negotiation.cc:407] Connection header received
0621 05:20:53.137544 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:53.137545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:53.137557 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:53.137607 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:53.138615 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.139390 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.140765 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.141077 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.141121 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.143727 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:53.143728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:53.143728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:53.143735 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:53.145448 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:53.145892 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:53.145893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:53.145894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:53.145941 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:53.146243 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:53.146243 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:53.146244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:53.146381 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:53.146428 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:53.148112 (+ 1684us) server_negotiation.cc:299] Negotiation successful
0621 05:20:53.148189 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 05:20:53.148447 9526 heartbeater.cc:347] Connected to a master server at 127.2.32.61:38731
I20260621 05:20:53.148535 9526 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:53.148537 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.136611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:43273)
0621 05:20:53.137000 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:53.137029 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:53.137431 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:53.137675 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:53.137679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:53.138272 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:20:53.138506 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:53.138512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.139458 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.139458 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:53.140650 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:53.140653 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.141205 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.141206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:53.141271 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.142059 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:20:53.142064 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:20:53.143636 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:20:53.145525 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:53.145526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:53.145526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:53.145792 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:53.146019 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:20:53.146020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:20:53.146021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:20:53.146161 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:20:53.146656 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:20:53.146662 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:20:53.146819 (+ 157us) client_negotiation.cc:769] Sending connection context
0621 05:20:53.148311 (+ 1492us) client_negotiation.cc:240] Negotiation successful
0621 05:20:53.148385 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":81,"thread_start_us":90,"threads_started":1}
I20260621 05:20:53.148718 9526 heartbeater.cc:511] Master 127.2.32.61:38731 requested a full tablet report, sending...
I20260621 05:20:53.149320 9524 heartbeater.cc:347] Connected to a master server at 127.2.32.60:37877
I20260621 05:20:53.149298 9525 heartbeater.cc:347] Connected to a master server at 127.2.32.62:45421
I20260621 05:20:53.149399 9524 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:53.149412 9525 heartbeater.cc:464] Registering TS with master...
I20260621 05:20:53.149441 8885 ts_manager.cc:194] Registered new tserver with Master: 73614dde1bc14e4889b823064cba9b95 (127.2.32.3:38567)
I20260621 05:20:53.149459 9524 heartbeater.cc:511] Master 127.2.32.60:37877 requested a full tablet report, sending...
I20260621 05:20:53.149478 9525 heartbeater.cc:511] Master 127.2.32.62:45421 requested a full tablet report, sending...
I20260621 05:20:53.149731 8281 ts_manager.cc:194] Re-registered known tserver with Master: 73614dde1bc14e4889b823064cba9b95 (127.2.32.3:38567)
I20260621 05:20:53.149752 9064 ts_manager.cc:194] Registered new tserver with Master: 73614dde1bc14e4889b823064cba9b95 (127.2.32.3:38567)
I20260621 05:20:53.150091 8281 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:43273
I20260621 05:20:53.155689 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.146977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44738 (local address 127.2.32.61:38731)
0621 05:20:53.147387 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:53.147389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:53.147701 (+ 312us) server_negotiation.cc:407] Connection header received
0621 05:20:53.147795 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:53.147796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:53.147804 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:53.147830 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:53.148603 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.149309 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.150676 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.150966 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.151037 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.152969 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:53.152970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:53.152970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:53.152977 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:53.154530 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:53.154914 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:53.154915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:53.154916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:53.154974 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:53.155216 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:53.155217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:53.155218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:53.155324 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:53.155427 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:53.155528 (+ 101us) server_negotiation.cc:299] Negotiation successful
0621 05:20:53.155597 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":52,"threads_started":1}
I20260621 05:20:53.157686 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.147352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35390 (local address 127.2.32.60:37877)
0621 05:20:53.147541 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:53.147543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:53.150692 (+ 3149us) server_negotiation.cc:407] Connection header received
0621 05:20:53.150753 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:53.150753 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:53.150762 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:53.150806 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:53.151496 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.151946 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.152994 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.153186 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.153258 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.154686 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:53.154687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:53.154688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:53.154694 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:53.156392 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:53.156869 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:53.156870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:53.156870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:53.156938 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:53.157166 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:53.157166 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:53.157167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:53.157289 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:53.157383 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:53.157569 (+ 186us) server_negotiation.cc:299] Negotiation successful
0621 05:20:53.157634 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":61,"threads_started":1}
I20260621 05:20:53.157891 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.146589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48052 (local address 127.2.32.62:45421)
0621 05:20:53.146876 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:53.146878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:53.146886 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:20:53.146905 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:53.146906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:53.146918 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:53.146981 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:53.150621 (+ 3640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.151354 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.152058 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.152250 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.152324 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.153853 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:53.153855 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:53.153855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:53.153862 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:53.156455 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:53.156819 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:53.156820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:53.156820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:53.156881 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:53.157094 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:53.157095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:53.157096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:53.157216 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:53.157292 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:53.157509 (+ 217us) server_negotiation.cc:299] Negotiation successful
0621 05:20:53.157802 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":72,"threads_started":1}
I20260621 05:20:53.159528 8281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:20:53.161620 8281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:20:53.169252 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.165293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:38567 (local address 127.0.0.1:56132)
0621 05:20:53.165367 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:53.165378 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:53.165415 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:53.166457 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:53.166458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:53.166459 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:53.166491 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:53.166493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.167503 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.167504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:53.168378 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:53.168380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.169034 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.169035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:53.169091 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.169094 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:20:53.169135 (+ 41us) client_negotiation.cc:240] Negotiation successful
0621 05:20:53.169184 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":1}
I20260621 05:20:53.169649 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.165503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56132 (local address 127.2.32.3:38567)
0621 05:20:53.166283 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:53.166286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:53.166309 (+ 23us) server_negotiation.cc:407] Connection header received
0621 05:20:53.166347 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:53.166349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:53.166365 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:53.166441 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:53.166578 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.167423 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.168483 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.168951 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.169035 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.169528 (+ 493us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:53.169564 (+ 36us) server_negotiation.cc:299] Negotiation successful
0621 05:20:53.169596 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":69,"threads_started":1}
I20260621 05:20:53.170447 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.165397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:42265 (local address 127.0.0.1:59648)
0621 05:20:53.165686 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:53.165694 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:53.165738 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:53.166718 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:53.166718 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:53.166721 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:53.166754 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:53.166755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.167970 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.167971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:53.169203 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:53.169206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.170195 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.170196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:53.170258 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.170261 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:20:53.170332 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 05:20:53.170373 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":40,"thread_start_us":108,"threads_started":1}
I20260621 05:20:53.170356 9458 tablet_service.cc:1467] Processing CreateTablet for tablet afb9513373954a01820c8d5257c61ce9 (DEFAULT_TABLE table=test-table [id=dfea3443b613487f9d8227b4ddd5417b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:53.170718 9458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb9513373954a01820c8d5257c61ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:53.170749 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.165485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59648 (local address 127.2.32.1:42265)
0621 05:20:53.166551 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:53.166555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:53.166566 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:20:53.166609 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:53.166611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:53.166627 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:53.166710 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:53.166818 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.167874 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.169320 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.170021 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.170094 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.170159 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:53.170606 (+ 447us) server_negotiation.cc:299] Negotiation successful
0621 05:20:53.170651 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1031,"thread_start_us":77,"threads_started":1}
I20260621 05:20:53.171231 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.165591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:38853 (local address 127.0.0.1:54944)
0621 05:20:53.165906 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:53.165915 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:53.166128 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:53.166448 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:53.166448 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:53.166451 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:53.166532 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:53.166534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.167971 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.167972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:53.169203 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:53.169206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.170955 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.170957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:53.171023 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.171028 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:20:53.171101 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 05:20:53.171166 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":74,"thread_start_us":62,"threads_started":1}
I20260621 05:20:53.171270 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.165854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.2.32.2:38853)
0621 05:20:53.166204 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:53.166208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:53.166219 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:20:53.166265 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:53.166266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:53.166286 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:53.166376 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:53.166625 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.167761 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.169313 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.170497 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.170567 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.170916 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:53.171119 (+ 203us) server_negotiation.cc:299] Negotiation successful
0621 05:20:53.171188 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":110,"threads_started":1}
I20260621 05:20:53.171482 9185 tablet_service.cc:1467] Processing CreateTablet for tablet afb9513373954a01820c8d5257c61ce9 (DEFAULT_TABLE table=test-table [id=dfea3443b613487f9d8227b4ddd5417b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:53.171785 9185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb9513373954a01820c8d5257c61ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:53.171875 9322 tablet_service.cc:1467] Processing CreateTablet for tablet afb9513373954a01820c8d5257c61ce9 (DEFAULT_TABLE table=test-table [id=dfea3443b613487f9d8227b4ddd5417b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:20:53.172112 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb9513373954a01820c8d5257c61ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:20:53.174537 9550 tablet_bootstrap.cc:492] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95: Bootstrap starting.
I20260621 05:20:53.175334 9550 tablet_bootstrap.cc:654] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:53.175380 9551 tablet_bootstrap.cc:492] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5: Bootstrap starting.
I20260621 05:20:53.175671 9550 log.cc:824] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:53.176084 9551 tablet_bootstrap.cc:654] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:53.176414 9551 log.cc:824] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:53.176574 9550 tablet_bootstrap.cc:492] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95: No bootstrap required, opened a new log
I20260621 05:20:53.176646 9550 ts_tablet_manager.cc:1397] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:20:53.176733 9552 tablet_bootstrap.cc:492] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1: Bootstrap starting.
I20260621 05:20:53.177161 9551 tablet_bootstrap.cc:492] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5: No bootstrap required, opened a new log
I20260621 05:20:53.177230 9551 ts_tablet_manager.cc:1397] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:20:53.179172 9550 raft_consensus.cc:348] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73614dde1bc14e4889b823064cba9b95" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38567 } } peers { permanent_uuid: "4b7d4becc8c54574807da6bc9cddabf1" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 42265 } } peers { permanent_uuid: "533996b4b53d4acb81ac269071c0adc5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38853 } }
I20260621 05:20:53.179337 9550 raft_consensus.cc:374] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:53.179378 9550 raft_consensus.cc:729] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73614dde1bc14e4889b823064cba9b95, State: Initialized, Role: FOLLOWER
I20260621 05:20:53.179513 9550 consensus_queue.cc:260] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73614dde1bc14e4889b823064cba9b95" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38567 } } peers { permanent_uuid: "4b7d4becc8c54574807da6bc9cddabf1" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 42265 } } peers { permanent_uuid: "533996b4b53d4acb81ac269071c0adc5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38853 } }
I20260621 05:20:53.179718 9552 tablet_bootstrap.cc:654] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1: Neither blocks nor log segments found. Creating new log.
I20260621 05:20:53.179769 9550 ts_tablet_manager.cc:1428] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:20:53.179776 9551 raft_consensus.cc:348] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73614dde1bc14e4889b823064cba9b95" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38567 } } peers { permanent_uuid: "4b7d4becc8c54574807da6bc9cddabf1" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 42265 } } peers { permanent_uuid: "533996b4b53d4acb81ac269071c0adc5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38853 } }
I20260621 05:20:53.179900 9525 heartbeater.cc:503] Master 127.2.32.62:45421 was elected leader, sending a full tablet report...
I20260621 05:20:53.180006 9551 raft_consensus.cc:374] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:53.180079 9551 raft_consensus.cc:729] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533996b4b53d4acb81ac269071c0adc5, State: Initialized, Role: FOLLOWER
I20260621 05:20:53.180110 9552 log.cc:824] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1: Log is configured to *not* fsync() on all Append() calls
I20260621 05:20:53.180224 9551 consensus_queue.cc:260] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73614dde1bc14e4889b823064cba9b95" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38567 } } peers { permanent_uuid: "4b7d4becc8c54574807da6bc9cddabf1" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 42265 } } peers { permanent_uuid: "533996b4b53d4acb81ac269071c0adc5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38853 } }
I20260621 05:20:53.180495 9551 ts_tablet_manager.cc:1428] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260621 05:20:53.180900 9389 heartbeater.cc:503] Master 127.2.32.62:45421 was elected leader, sending a full tablet report...
I20260621 05:20:53.181800 9552 tablet_bootstrap.cc:492] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1: No bootstrap required, opened a new log
I20260621 05:20:53.181874 9552 ts_tablet_manager.cc:1397] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260621 05:20:53.183707 9552 raft_consensus.cc:348] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73614dde1bc14e4889b823064cba9b95" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38567 } } peers { permanent_uuid: "4b7d4becc8c54574807da6bc9cddabf1" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 42265 } } peers { permanent_uuid: "533996b4b53d4acb81ac269071c0adc5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38853 } }
I20260621 05:20:53.183816 9552 raft_consensus.cc:374] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:20:53.183836 9552 raft_consensus.cc:729] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7d4becc8c54574807da6bc9cddabf1, State: Initialized, Role: FOLLOWER
I20260621 05:20:53.183934 9552 consensus_queue.cc:260] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73614dde1bc14e4889b823064cba9b95" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38567 } } peers { permanent_uuid: "4b7d4becc8c54574807da6bc9cddabf1" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 42265 } } peers { permanent_uuid: "533996b4b53d4acb81ac269071c0adc5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38853 } }
I20260621 05:20:53.184149 9552 ts_tablet_manager.cc:1428] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:20:53.184301 9252 heartbeater.cc:503] Master 127.2.32.62:45421 was elected leader, sending a full tablet report...
I20260621 05:20:53.206591 9558 raft_consensus.cc:482] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:20:53.206727 9558 raft_consensus.cc:504] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73614dde1bc14e4889b823064cba9b95" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38567 } } peers { permanent_uuid: "4b7d4becc8c54574807da6bc9cddabf1" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 42265 } } peers { permanent_uuid: "533996b4b53d4acb81ac269071c0adc5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38853 } }
I20260621 05:20:53.207001 9558 leader_election.cc:288] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73614dde1bc14e4889b823064cba9b95 (127.2.32.3:38567), 533996b4b53d4acb81ac269071c0adc5 (127.2.32.2:38853)
I20260621 05:20:53.209587 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.207123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:38567 (local address 127.2.32.1:34579)
0621 05:20:53.207179 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:53.207191 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:53.207232 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:53.207350 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:53.207350 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:53.207351 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:53.207423 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:53.207425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.208041 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.208042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:53.208799 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:53.208801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.209318 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.209319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:53.209388 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.209391 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:20:53.209469 (+ 78us) client_negotiation.cc:240] Negotiation successful
0621 05:20:53.209504 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260621 05:20:53.209587 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.207193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:34579 (local address 127.2.32.3:38567)
0621 05:20:53.207240 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:53.207242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:53.207252 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:20:53.207286 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:53.207287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:53.207296 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:53.207338 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:53.207500 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.207961 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.208887 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.209237 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.209319 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.209330 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:53.209477 (+ 147us) server_negotiation.cc:299] Negotiation successful
0621 05:20:53.209514 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260621 05:20:53.209734 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.207241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:45095 (local address 127.2.32.2:38853)
0621 05:20:53.207290 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:53.207292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:53.207324 (+ 32us) server_negotiation.cc:407] Connection header received
0621 05:20:53.207411 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:53.207412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:53.207424 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:53.207505 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:20:53.207647 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.208093 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.209037 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.209380 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.209456 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.209465 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:53.209641 (+ 176us) server_negotiation.cc:299] Negotiation successful
0621 05:20:53.209674 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260621 05:20:53.209749 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.207220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:38853 (local address 127.2.32.1:45095)
0621 05:20:53.207275 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:53.207286 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:20:53.207338 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:20:53.207515 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:20:53.207515 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:20:53.207519 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:20:53.207580 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:53.207581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.208175 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.208176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:53.208931 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:20:53.208933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.209458 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.209459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:20:53.209567 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.209570 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:20:53.209637 (+ 67us) client_negotiation.cc:240] Negotiation successful
0621 05:20:53.209673 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260621 05:20:53.210093 9478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb9513373954a01820c8d5257c61ce9" candidate_uuid: "4b7d4becc8c54574807da6bc9cddabf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73614dde1bc14e4889b823064cba9b95" is_pre_election: true
I20260621 05:20:53.210225 9478 raft_consensus.cc:2454] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b7d4becc8c54574807da6bc9cddabf1 in term 0.
I20260621 05:20:53.210208 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb9513373954a01820c8d5257c61ce9" candidate_uuid: "4b7d4becc8c54574807da6bc9cddabf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533996b4b53d4acb81ac269071c0adc5" is_pre_election: true
I20260621 05:20:53.210342 9342 raft_consensus.cc:2454] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b7d4becc8c54574807da6bc9cddabf1 in term 0.
I20260621 05:20:53.210429 9139 leader_election.cc:302] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7d4becc8c54574807da6bc9cddabf1, 73614dde1bc14e4889b823064cba9b95; no voters:
I20260621 05:20:53.210575 9558 raft_consensus.cc:2781] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:20:53.210625 9558 raft_consensus.cc:482] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:20:53.210640 9558 raft_consensus.cc:3037] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:53.211246 9558 raft_consensus.cc:504] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73614dde1bc14e4889b823064cba9b95" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38567 } } peers { permanent_uuid: "4b7d4becc8c54574807da6bc9cddabf1" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 42265 } } peers { permanent_uuid: "533996b4b53d4acb81ac269071c0adc5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38853 } }
I20260621 05:20:53.211376 9558 leader_election.cc:288] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [CANDIDATE]: Term 1 election: Requested vote from peers 73614dde1bc14e4889b823064cba9b95 (127.2.32.3:38567), 533996b4b53d4acb81ac269071c0adc5 (127.2.32.2:38853)
I20260621 05:20:53.211514 9478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb9513373954a01820c8d5257c61ce9" candidate_uuid: "4b7d4becc8c54574807da6bc9cddabf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73614dde1bc14e4889b823064cba9b95"
I20260621 05:20:53.211562 9342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb9513373954a01820c8d5257c61ce9" candidate_uuid: "4b7d4becc8c54574807da6bc9cddabf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533996b4b53d4acb81ac269071c0adc5"
I20260621 05:20:53.211607 9478 raft_consensus.cc:3037] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:53.211645 9342 raft_consensus.cc:3037] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:20:53.212283 9342 raft_consensus.cc:2454] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7d4becc8c54574807da6bc9cddabf1 in term 1.
I20260621 05:20:53.212327 9478 raft_consensus.cc:2454] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7d4becc8c54574807da6bc9cddabf1 in term 1.
I20260621 05:20:53.212857 9140 leader_election.cc:302] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7d4becc8c54574807da6bc9cddabf1, 533996b4b53d4acb81ac269071c0adc5; no voters:
I20260621 05:20:53.212976 9558 raft_consensus.cc:2781] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:20:53.213119 9558 raft_consensus.cc:686] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [term 1 LEADER]: Becoming Leader. State: Replica: 4b7d4becc8c54574807da6bc9cddabf1, State: Running, Role: LEADER
I20260621 05:20:53.213210 9558 consensus_queue.cc:237] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73614dde1bc14e4889b823064cba9b95" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38567 } } peers { permanent_uuid: "4b7d4becc8c54574807da6bc9cddabf1" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 42265 } } peers { permanent_uuid: "533996b4b53d4acb81ac269071c0adc5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38853 } }
I20260621 05:20:53.214020 8281 catalog_manager.cc:5495] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b7d4becc8c54574807da6bc9cddabf1 (127.2.32.1). New cstate: current_term: 1 leader_uuid: "4b7d4becc8c54574807da6bc9cddabf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73614dde1bc14e4889b823064cba9b95" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7d4becc8c54574807da6bc9cddabf1" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 42265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533996b4b53d4acb81ac269071c0adc5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38853 } health_report { overall_health: UNKNOWN } } }
I20260621 05:20:53.226768 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.224302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.2.32.1:42265)
0621 05:20:53.224351 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:53.224353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:53.224363 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:20:53.224416 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:53.224417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:53.224443 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:53.224532 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:20:53.224683 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.225213 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.225974 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.226179 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.226269 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.226456 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:20:53.226549 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:20:53.226647 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:53.226671 (+ 24us) server_negotiation.cc:299] Negotiation successful
0621 05:20:53.226691 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:20:53.228905 9342 raft_consensus.cc:1261] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5 [term 1 FOLLOWER]: Refusing update from remote peer 4b7d4becc8c54574807da6bc9cddabf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:53.229010 9478 raft_consensus.cc:1261] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95 [term 1 FOLLOWER]: Refusing update from remote peer 4b7d4becc8c54574807da6bc9cddabf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:20:53.229266 9558 consensus_queue.cc:1035] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533996b4b53d4acb81ac269071c0adc5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:53.229434 9559 consensus_queue.cc:1035] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73614dde1bc14e4889b823064cba9b95" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:20:53.232339 9563 mvcc.cc:205] Tried to move back new op lower bound from 7299150861222957056 to 7299150861161934848. Current Snapshot: MvccSnapshot[applied={T|T < 7299150861222957056}]
I20260621 05:20:53.235225 8281 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:20:53.235334 8281 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:20:53.237102 8281 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 087a170fca294b98b94961bab595c623: Sending DeleteTablet for 3 replicas of tablet afb9513373954a01820c8d5257c61ce9
I20260621 05:20:53.237524 9458 tablet_service.cc:1514] Processing DeleteTablet for tablet afb9513373954a01820c8d5257c61ce9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:53 UTC) from {username='oryx', principal='oryx/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:56132
I20260621 05:20:53.237772 9322 tablet_service.cc:1514] Processing DeleteTablet for tablet afb9513373954a01820c8d5257c61ce9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:53 UTC) from {username='oryx', principal='oryx/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:54944
I20260621 05:20:53.237811 9185 tablet_service.cc:1514] Processing DeleteTablet for tablet afb9513373954a01820c8d5257c61ce9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:20:53 UTC) from {username='oryx', principal='oryx/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:59648
I20260621 05:20:53.238014 9574 tablet_replica.cc:330] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5: stopping tablet replica
I20260621 05:20:53.238119 9574 raft_consensus.cc:2229] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:20:53.238222 9574 raft_consensus.cc:2258] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:53.238762 9574 ts_tablet_manager.cc:1905] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:53.239215 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 8249
I20260621 05:20:53.240774 9574 ts_tablet_manager.cc:1918] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:20:53.240917 9574 log.cc:1196] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/afb9513373954a01820c8d5257c61ce9
I20260621 05:20:53.241266 9574 ts_tablet_manager.cc:1939] T afb9513373954a01820c8d5257c61ce9 P 533996b4b53d4acb81ac269071c0adc5: Deleting consensus metadata
I20260621 05:20:53.241554 9575 tablet_replica.cc:330] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1: stopping tablet replica
I20260621 05:20:53.241664 9575 raft_consensus.cc:2229] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:20:53.241751 9575 raft_consensus.cc:2258] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:53.242236 9575 ts_tablet_manager.cc:1905] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:53.244211 9575 ts_tablet_manager.cc:1918] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:20:53.244290 9575 log.cc:1196] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-0/wal/wals/afb9513373954a01820c8d5257c61ce9
I20260621 05:20:53.244557 9575 ts_tablet_manager.cc:1939] T afb9513373954a01820c8d5257c61ce9 P 4b7d4becc8c54574807da6bc9cddabf1: Deleting consensus metadata
I20260621 05:20:53.241901 9573 tablet_replica.cc:330] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95: stopping tablet replica
I20260621 05:20:53.245079 9573 raft_consensus.cc:2229] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:20:53.245206 9573 raft_consensus.cc:2258] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:20:53.245671 9573 ts_tablet_manager.cc:1905] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:20:53.247306 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.238372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35400 (local address 127.2.32.60:37877)
0621 05:20:53.238444 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:53.238446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:53.238607 (+ 161us) server_negotiation.cc:407] Connection header received
0621 05:20:53.238705 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:53.238706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:53.238716 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:53.238747 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:53.239952 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.240814 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.241920 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.242206 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.242254 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.244198 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:53.244200 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:53.244200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:53.244207 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:53.245971 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:53.246413 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:53.246414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:53.246414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:53.246472 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:53.246688 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:53.246689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:53.246689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:53.246829 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:53.246906 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:53.247112 (+ 206us) server_negotiation.cc:299] Negotiation successful
0621 05:20:53.247208 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260621 05:20:53.247370 9275 connection.cc:535] server connection from 127.0.0.1:54944 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 05:20:53.247471 9573 ts_tablet_manager.cc:1918] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:20:53.247529 9573 log.cc:1196] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/afb9513373954a01820c8d5257c61ce9
I20260621 05:20:53.247787 9573 ts_tablet_manager.cc:1939] T afb9513373954a01820c8d5257c61ce9 P 73614dde1bc14e4889b823064cba9b95: Deleting consensus metadata
W20260621 05:20:53.247788 9138 connection.cc:535] server connection from 127.0.0.1:59648 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260621 05:20:53.248162 9412 connection.cc:441] server connection from 127.0.0.1:56132 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56132 (request call id 1) could send its response
I20260621 05:20:53.258019 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.249890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35404 (local address 127.2.32.60:37877)
0621 05:20:53.249925 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:53.249927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:53.250073 (+ 146us) server_negotiation.cc:407] Connection header received
0621 05:20:53.250162 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:53.250163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:53.250173 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:53.250203 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:53.250887 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.251656 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.253189 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.253479 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.253544 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.255092 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:53.255093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:53.255094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:53.255100 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:53.256751 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:53.257171 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:53.257172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:53.257172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:53.257233 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:53.257468 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:53.257469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:53.257469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:53.257584 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:53.257686 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:53.257828 (+ 142us) server_negotiation.cc:299] Negotiation successful
0621 05:20:53.257903 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:20:53.258260 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:53.249716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.2.32.61:38731)
0621 05:20:53.249765 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:20:53.249767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:20:53.249982 (+ 215us) server_negotiation.cc:407] Connection header received
0621 05:20:53.250068 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:20:53.250069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:20:53.250080 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:20:53.250135 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:20:53.251215 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.251670 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.252740 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:20:53.253060 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:20:53.253136 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:20:53.255085 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:20:53.255087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:20:53.255088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:20:53.255094 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:20:53.256863 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:53.257288 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:53.257289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:53.257289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:53.257355 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:20:53.257576 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:20:53.257577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:20:53.257578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:20:53.257704 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:20:53.257772 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 05:20:53.257977 (+ 205us) server_negotiation.cc:299] Negotiation successful
0621 05:20:53.258063 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:20:54.233517 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:54.233226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:35667)
0621 05:20:54.233350 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:54.233449 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
W20260621 05:20:54.233769 9252 heartbeater.cc:644] Failed to heartbeat to 127.2.32.62:45421 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
I20260621 05:20:54.233994 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:54.233540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:34399)
0621 05:20:54.233781 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:54.233952 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":46,"threads_started":1}
W20260621 05:20:54.234135 9525 heartbeater.cc:644] Failed to heartbeat to 127.2.32.62:45421 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
I20260621 05:20:54.234383 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:54.234001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:46651)
0621 05:20:54.234248 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:54.234315 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":47,"threads_started":1}
W20260621 05:20:54.234584 9389 heartbeater.cc:644] Failed to heartbeat to 127.2.32.62:45421 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
I20260621 05:20:54.737771 9587 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 087a170fca294b98b94961bab595c623)
I20260621 05:20:54.738008 9587 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: false } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } }
I20260621 05:20:54.738543 9587 leader_election.cc:288] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 087a170fca294b98b94961bab595c623 (127.2.32.62:45421)
I20260621 05:20:54.739027 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:54.738805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.0.0.1:48480)
0621 05:20:54.738947 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:54.738989 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
W20260621 05:20:54.739190 8872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
I20260621 05:20:54.739861 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:54.739634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.0.0.1:48496)
0621 05:20:54.739785 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:54.739823 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260621 05:20:54.739984 8872 leader_election.cc:334] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 087a170fca294b98b94961bab595c623 (127.2.32.62:45421): Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
I20260621 05:20:54.740039 8872 leader_election.cc:302] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f0198561afcd4b85a28fa23b00c61a95; no voters: 087a170fca294b98b94961bab595c623
I20260621 05:20:54.740227 9587 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:20:55.234724 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:55.234395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:40791)
0621 05:20:55.234529 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:55.234671 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":48,"threads_started":1}
I20260621 05:20:55.234752 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:55.234256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:42195)
0621 05:20:55.234586 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:55.234625 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":83,"threads_started":1}
I20260621 05:20:55.235268 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:55.234827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:33535)
0621 05:20:55.235186 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:55.235220 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":69,"threads_started":1}
I20260621 05:20:56.235462 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:56.235226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:39473)
0621 05:20:56.235363 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:56.235403 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260621 05:20:56.235965 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:56.235426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:49089)
0621 05:20:56.235836 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:56.235899 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":187,"threads_started":1}
I20260621 05:20:56.236196 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:56.235830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:57213)
0621 05:20:56.236112 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:56.236148 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":51,"threads_started":1}
I20260621 05:20:56.335546 9596 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 087a170fca294b98b94961bab595c623)
I20260621 05:20:56.335652 9596 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: false } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } }
I20260621 05:20:56.335832 9596 leader_election.cc:288] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 087a170fca294b98b94961bab595c623 (127.2.32.62:45421)
I20260621 05:20:56.336300 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:56.335981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.0.0.1:48730)
0621 05:20:56.336171 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:56.336246 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":61,"threads_started":1}
I20260621 05:20:56.336876 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:56.336754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.0.0.1:48738)
0621 05:20:56.336809 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:56.336846 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260621 05:20:56.336980 8872 leader_election.cc:334] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 087a170fca294b98b94961bab595c623 (127.2.32.62:45421): Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
I20260621 05:20:56.337062 8872 leader_election.cc:302] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f0198561afcd4b85a28fa23b00c61a95; no voters: 087a170fca294b98b94961bab595c623
I20260621 05:20:56.337255 9596 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:20:57.236546 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:57.236273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:55371)
0621 05:20:57.236449 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:57.236492 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":71,"threads_started":1}
I20260621 05:20:57.237187 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:57.236899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:52355)
0621 05:20:57.237066 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:57.237114 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":76,"threads_started":1}
I20260621 05:20:57.237468 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:57.236679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:52893)
0621 05:20:57.237373 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:57.237412 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":80,"threads_started":1}
I20260621 05:20:58.237809 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:58.237307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:33119)
0621 05:20:58.237699 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:58.237745 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":99,"threads_started":1}
I20260621 05:20:58.238034 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:58.237849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:40541)
0621 05:20:58.237947 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:58.237984 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":53,"threads_started":1}
I20260621 05:20:58.238276 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:58.237941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:40589)
0621 05:20:58.238193 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:58.238232 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":62,"threads_started":1}
I20260621 05:20:58.353582 9605 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 087a170fca294b98b94961bab595c623)
I20260621 05:20:58.353806 9605 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: false } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } }
I20260621 05:20:58.354013 9605 leader_election.cc:288] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 087a170fca294b98b94961bab595c623 (127.2.32.62:45421)
I20260621 05:20:58.354451 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:58.354163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.0.0.1:41764)
0621 05:20:58.354330 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:58.354396 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":50,"threads_started":1}
I20260621 05:20:58.355096 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:58.354904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.0.0.1:41772)
0621 05:20:58.355019 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:58.355065 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260621 05:20:58.355269 8872 leader_election.cc:334] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 087a170fca294b98b94961bab595c623 (127.2.32.62:45421): Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
I20260621 05:20:58.355312 8872 leader_election.cc:302] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f0198561afcd4b85a28fa23b00c61a95; no voters: 087a170fca294b98b94961bab595c623
I20260621 05:20:58.355386 9605 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:20:59.238852 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:59.238542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:56127)
0621 05:20:59.238752 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:59.238791 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":80,"threads_started":1}
I20260621 05:20:59.239046 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:59.238767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:52033)
0621 05:20:59.238958 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:59.238998 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":55,"threads_started":1}
I20260621 05:20:59.239269 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:20:59.238930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:55219)
0621 05:20:59.239186 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0621 05:20:59.239221 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":52,"threads_started":1}
I20260621 05:21:00.239815 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:00.239404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:50777)
0621 05:21:00.239711 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:00.239752 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":64,"threads_started":1}
I20260621 05:21:00.240173 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:00.239904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:60717)
0621 05:21:00.240088 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:00.240127 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":69,"threads_started":1}
I20260621 05:21:00.240540 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:00.240100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:43903)
0621 05:21:00.240462 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:00.240502 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":59,"threads_started":1}
I20260621 05:21:01.240993 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:01.240552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:43105)
0621 05:21:01.240926 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:01.240954 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":68,"threads_started":1}
I20260621 05:21:01.240970 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:01.240704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:44833)
0621 05:21:01.240867 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:01.240913 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":57,"threads_started":1}
I20260621 05:21:01.241209 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:01.240892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:42939)
0621 05:21:01.241045 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:01.241153 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":50,"threads_started":1}
I20260621 05:21:02.241690 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:02.241377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:54797)
0621 05:21:02.241580 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:02.241619 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260621 05:21:02.241886 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:02.241663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:38947)
0621 05:21:02.241806 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:02.241841 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
I20260621 05:21:02.242093 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:02.241577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:48571)
0621 05:21:02.242019 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:02.242051 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":67,"threads_started":1}
I20260621 05:21:02.435985 9620 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 087a170fca294b98b94961bab595c623)
I20260621 05:21:02.436121 9620 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: false } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } }
I20260621 05:21:02.436322 9620 leader_election.cc:288] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 087a170fca294b98b94961bab595c623 (127.2.32.62:45421)
I20260621 05:21:02.436792 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:02.436572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.0.0.1:42082)
0621 05:21:02.436703 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:02.436754 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":88,"threads_started":1}
W20260621 05:21:02.436888 8872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260621 05:21:02.437247 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:02.437150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.0.0.1:42096)
0621 05:21:02.437201 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:02.437229 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260621 05:21:02.437335 8872 leader_election.cc:334] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 087a170fca294b98b94961bab595c623 (127.2.32.62:45421): Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
I20260621 05:21:02.437387 8872 leader_election.cc:302] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f0198561afcd4b85a28fa23b00c61a95; no voters: 087a170fca294b98b94961bab595c623
I20260621 05:21:02.437481 9620 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:21:03.242937 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:03.242551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:57645)
0621 05:21:03.242830 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:03.242878 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":67,"threads_started":1}
I20260621 05:21:03.242940 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:03.242303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:50263)
0621 05:21:03.242871 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:03.242905 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":82,"threads_started":1}
I20260621 05:21:03.242914 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:03.242333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:58839)
0621 05:21:03.242774 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:03.242835 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":75,"threads_started":1}
I20260621 05:21:04.243804 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:04.243418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:57131)
0621 05:21:04.243698 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:04.243743 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":77,"threads_started":1}
I20260621 05:21:04.243858 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:04.243421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:50545)
0621 05:21:04.243772 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:04.243817 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":64,"threads_started":1}
I20260621 05:21:04.244201 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:04.243659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:45447)
0621 05:21:04.244126 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:04.244160 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":50,"threads_started":1}
I20260621 05:21:05.245111 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:05.244533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:55923)
0621 05:21:05.244905 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:05.245033 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":89,"threads_started":1}
I20260621 05:21:05.245229 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:05.244762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:39339)
0621 05:21:05.245099 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:05.245136 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":52,"threads_started":1}
I20260621 05:21:05.245520 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:05.244533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:42073)
0621 05:21:05.245380 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:05.245466 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":826,"thread_start_us":99,"threads_started":1}
I20260621 05:21:06.246385 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:06.246075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:57231)
0621 05:21:06.246253 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:06.246302 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":71,"threads_started":1}
I20260621 05:21:06.246374 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:06.246032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:59443)
0621 05:21:06.246253 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:06.246301 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":113,"threads_started":1}
I20260621 05:21:06.246652 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:06.246381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:46815)
0621 05:21:06.246575 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:06.246612 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
I20260621 05:21:07.247291 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:07.246934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:53759)
0621 05:21:07.247179 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:07.247225 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":72,"threads_started":1}
I20260621 05:21:07.247632 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:07.247236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:50187)
0621 05:21:07.247519 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:07.247566 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":107,"threads_started":1}
I20260621 05:21:07.247671 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:07.247328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:46615)
0621 05:21:07.247570 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:07.247612 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":60,"threads_started":1}
I20260621 05:21:07.672844 9638 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 087a170fca294b98b94961bab595c623)
I20260621 05:21:07.673031 9638 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: false } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } }
I20260621 05:21:07.673270 9638 leader_election.cc:288] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 087a170fca294b98b94961bab595c623 (127.2.32.62:45421)
I20260621 05:21:07.673758 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:07.673506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.0.0.1:42406)
0621 05:21:07.673661 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:07.673707 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
W20260621 05:21:07.673866 8872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
I20260621 05:21:07.674252 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:07.674157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.0.0.1:42408)
0621 05:21:07.674206 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:07.674230 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260621 05:21:07.674324 8872 leader_election.cc:334] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 087a170fca294b98b94961bab595c623 (127.2.32.62:45421): Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
I20260621 05:21:07.674357 8872 leader_election.cc:302] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f0198561afcd4b85a28fa23b00c61a95; no voters: 087a170fca294b98b94961bab595c623
I20260621 05:21:07.674443 9638 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:21:08.248716 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:08.248197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:41283)
0621 05:21:08.248592 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:08.248654 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":172,"threads_started":1}
I20260621 05:21:08.248811 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:08.248520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:44119)
0621 05:21:08.248726 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:08.248763 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":72,"threads_started":1}
I20260621 05:21:08.248569 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:08.247956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:39457)
0621 05:21:08.248291 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:08.248350 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":144,"threads_started":1}
I20260621 05:21:09.249679 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:09.249223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:58323)
0621 05:21:09.249468 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:09.249519 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":82,"threads_started":1}
I20260621 05:21:09.249676 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:09.249387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:54905)
0621 05:21:09.249558 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:09.249599 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":98,"threads_started":1}
I20260621 05:21:09.250002 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:09.249388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:53799)
0621 05:21:09.249893 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:09.249951 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":83,"threads_started":1}
I20260621 05:21:10.250666 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:10.250342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:53763)
0621 05:21:10.250546 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:10.250595 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260621 05:21:10.250914 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:10.250500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:35327)
0621 05:21:10.250752 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:10.250844 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":56,"threads_started":1}
I20260621 05:21:10.251019 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:10.250230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:50941)
0621 05:21:10.250924 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:10.250964 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":93,"threads_started":1}
I20260621 05:21:11.251737 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:11.251418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:53883)
0621 05:21:11.251587 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:11.251638 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260621 05:21:11.251974 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:11.251650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:44197)
0621 05:21:11.251874 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:11.251917 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":54,"threads_started":1}
I20260621 05:21:11.252148 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:11.251418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:54023)
0621 05:21:11.252056 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:11.252095 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":88,"threads_started":1}
I20260621 05:21:12.252660 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:12.252355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:52271)
0621 05:21:12.252536 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:12.252585 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":100,"threads_started":1}
I20260621 05:21:12.252970 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:12.252403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:45727)
0621 05:21:12.252870 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:12.252916 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":85,"threads_started":1}
I20260621 05:21:12.252970 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:12.252547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:45543)
0621 05:21:12.252888 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:12.252916 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":95,"threads_started":1}
I20260621 05:21:13.253530 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:13.253245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:56541)
0621 05:21:13.253425 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:13.253468 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":86,"threads_started":1}
I20260621 05:21:13.253718 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:13.253504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:42941)
0621 05:21:13.253611 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:13.253651 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":52,"threads_started":1}
I20260621 05:21:13.253769 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:13.253416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:42167)
0621 05:21:13.253694 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:13.253722 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":54,"threads_started":1}
I20260621 05:21:14.254606 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:14.254235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:37185)
0621 05:21:14.254468 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:14.254522 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":92,"threads_started":1}
I20260621 05:21:14.254630 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:14.254329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:50385)
0621 05:21:14.254528 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:14.254567 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260621 05:21:14.254834 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:14.254554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:54509)
0621 05:21:14.254758 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:14.254789 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":51,"threads_started":1}
I20260621 05:21:15.255779 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:15.255258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:48091)
0621 05:21:15.255668 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:15.255713 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":96,"threads_started":1}
I20260621 05:21:15.255836 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:15.255599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:56559)
0621 05:21:15.255753 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:15.255788 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":48,"threads_started":1}
I20260621 05:21:15.256083 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:15.255660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:40645)
0621 05:21:15.256002 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:15.256036 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":56,"threads_started":1}
I20260621 05:21:16.256565 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:16.256218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:54367)
0621 05:21:16.256460 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:16.256507 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":88,"threads_started":1}
I20260621 05:21:16.256624 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:16.256372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:36557)
0621 05:21:16.256522 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:16.256553 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":46,"threads_started":1}
I20260621 05:21:16.256947 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:16.256749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:59641)
0621 05:21:16.256859 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:16.256893 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":49,"threads_started":1}
I20260621 05:21:17.257505 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:17.257078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:55973)
0621 05:21:17.257406 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:17.257447 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":67,"threads_started":1}
I20260621 05:21:17.257525 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:17.257293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:46191)
0621 05:21:17.257433 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:17.257475 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":47,"threads_started":1}
I20260621 05:21:17.257681 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:17.257487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:44577)
0621 05:21:17.257616 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:17.257645 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260621 05:21:18.139309 9671 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 087a170fca294b98b94961bab595c623)
I20260621 05:21:18.139413 9671 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "087a170fca294b98b94961bab595c623" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45421 } } peers { permanent_uuid: "f0198561afcd4b85a28fa23b00c61a95" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 38731 } attrs { promote: false } } peers { permanent_uuid: "3440bc55a78b48ddb9f87e38c8bcf22a" member_type: NON_VOTER last_known_addr { host: "127.2.32.60" port: 37877 } attrs { promote: true } }
I20260621 05:21:18.139773 9671 leader_election.cc:288] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 087a170fca294b98b94961bab595c623 (127.2.32.62:45421)
I20260621 05:21:18.139915 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:18.139697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.0.0.1:35834)
0621 05:21:18.139814 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:18.139852 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":50,"threads_started":1}
W20260621 05:21:18.140018 8872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
I20260621 05:21:18.140336 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:18.140249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.0.0.1:35846)
0621 05:21:18.140294 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:18.140322 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260621 05:21:18.140421 8872 leader_election.cc:334] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 087a170fca294b98b94961bab595c623 (127.2.32.62:45421): Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
I20260621 05:21:18.140470 8872 leader_election.cc:302] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f0198561afcd4b85a28fa23b00c61a95; no voters: 087a170fca294b98b94961bab595c623
I20260621 05:21:18.140535 9671 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f0198561afcd4b85a28fa23b00c61a95 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:21:18.258281 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:18.257984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:38635)
0621 05:21:18.258184 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:18.258226 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":67,"threads_started":1}
I20260621 05:21:18.258302 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:18.257981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:42543)
0621 05:21:18.258234 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:18.258263 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":76,"threads_started":1}
I20260621 05:21:18.258550 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:18.258205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:51377)
0621 05:21:18.258464 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:18.258501 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":75,"threads_started":1}
I20260621 05:21:19.259112 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:19.258885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:43041)
0621 05:21:19.259009 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:19.259051 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260621 05:21:19.259181 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:19.258940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:51119)
0621 05:21:19.259083 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:19.259117 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260621 05:21:19.259331 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:19.258648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:41951)
0621 05:21:19.259260 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:19.259295 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":69,"threads_started":1}
I20260621 05:21:20.260422 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:20.259932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:33443)
0621 05:21:20.260248 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:20.260323 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":70,"threads_started":1}
I20260621 05:21:20.260576 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:20.259668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:38771)
0621 05:21:20.260490 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:20.260533 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":802,"thread_start_us":96,"threads_started":1}
I20260621 05:21:20.260664 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:20.260438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:39381)
0621 05:21:20.260546 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:20.260587 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":53,"threads_started":1}
I20260621 05:21:21.261502 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:21.261297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:45831)
0621 05:21:21.261400 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:21.261444 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":48,"threads_started":1}
I20260621 05:21:21.261518 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:21.261006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:43057)
0621 05:21:21.261411 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:21.261468 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":65,"threads_started":1}
I20260621 05:21:21.261547 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:21.261223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:46691)
0621 05:21:21.261473 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:21.261504 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":82,"threads_started":1}
I20260621 05:21:22.262283 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:22.262076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:58701)
0621 05:21:22.262187 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:22.262223 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
I20260621 05:21:22.262686 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:22.261914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:48919)
0621 05:21:22.262608 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:22.262639 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":57,"threads_started":1}
I20260621 05:21:22.262916 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:22.261923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:57717)
0621 05:21:22.262803 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:22.262840 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":862,"thread_start_us":58,"threads_started":1}
I20260621 05:21:23.263195 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:23.262861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.1:33695)
0621 05:21:23.263015 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:23.263060 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
I20260621 05:21:23.263545 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:23.263275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.2:51793)
0621 05:21:23.263459 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:23.263496 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":51,"threads_started":1}
I20260621 05:21:23.263557 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:23.263343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45421 (local address 127.2.32.3:32805)
0621 05:21:23.263450 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:23.263482 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:45421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":51,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.2.32.62:45421,127.2.32.61:38731,127.2.32.60:37877, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.32.62:45421,127.2.32.61:38731,127.2.32.60:37877, num_attempts: 1)
I20260621 05:21:23.492482 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 9032
I20260621 05:21:23.499526 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 9259
I20260621 05:21:23.504354 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 9395
I20260621 05:21:23.508848 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 8745
I20260621 05:21:23.514395 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 8939
2026-06-21T05:21:23Z chronyd exiting
I20260621 05:21:23.534682 2176 test_util.cc:175] -----------------------------------------------
I20260621 05:21:23.534760 2176 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782019225111293-2176-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (33317 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:21:23 dist-test-slave-b357 krb5kdc[9694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:21:23 dist-test-slave-b357 krb5kdc[9694](info): set up 2 sockets
Jun 21 05:21:23 dist-test-slave-b357 krb5kdc[9694](info): commencing operation
krb5kdc: starting...
W20260621 05:21:24.579685 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:21:24 dist-test-slave-b357 krb5kdc[9694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:21:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:21:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.62.keytab.
Entry for principal oryx/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.62.keytab.
I20260621 05:21:24.707340 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:43427
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.62.keytab
--principal=oryx/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:43427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:21:24.789355 9707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:24.789611 9707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:24.789649 9707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:24.790764 9707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:21:24.790836 9707 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 05:21:24.790859 9707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:24.790879 9707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:24.790910 9707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:21:24.790932 9707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:21:24.792431 9707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:43427
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.62.keytab
--principal=oryx/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:43427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:24.792714 9707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:24.792946 9707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:24.797976 9713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:24.797986 9715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:24.797986 9712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:24.798135 9707 server_base.cc:1034] running on GCE node
I20260621 05:21:24.798473 9707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:24.798763 9707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:24.799918 9707 hybrid_clock.cc:648] HybridClock initialized: now 1782019284799899 us; error 48 us; skew 500 ppm
Jun 21 05:21:24 dist-test-slave-b357 krb5kdc[9694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019284, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:21:24.803398 9707 init.cc:377] Logged in from keytab as oryx/127.2.32.62@KRBTEST.COM (short username oryx)
I20260621 05:21:24.804059 9707 webserver.cc:460] Webserver started at http://127.2.32.62:46265/ using document root <none> and password file <none>
I20260621 05:21:24.804343 9707 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:24.804417 9707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:24.804591 9707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:21:24.805884 9707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "8aa2a63cb3f94a838d32185a18ab7fe9"
format_stamp: "Formatted at 2026-06-21 05:21:24 on dist-test-slave-b357"
server_key: "67edd43f51a3cff1d91ad4cf39dbd994"
server_key_iv: "654b08c61fa85fdc2dbad9aa1e1920ec"
server_key_version: "encryptionkey@0"
I20260621 05:21:24.806582 9707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "8aa2a63cb3f94a838d32185a18ab7fe9"
format_stamp: "Formatted at 2026-06-21 05:21:24 on dist-test-slave-b357"
server_key: "67edd43f51a3cff1d91ad4cf39dbd994"
server_key_iv: "654b08c61fa85fdc2dbad9aa1e1920ec"
server_key_version: "encryptionkey@0"
I20260621 05:21:24.808254 9707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260621 05:21:24.809370 9722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:24.809674 9707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:21:24.809877 9707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "8aa2a63cb3f94a838d32185a18ab7fe9"
format_stamp: "Formatted at 2026-06-21 05:21:24 on dist-test-slave-b357"
server_key: "67edd43f51a3cff1d91ad4cf39dbd994"
server_key_iv: "654b08c61fa85fdc2dbad9aa1e1920ec"
server_key_version: "encryptionkey@0"
I20260621 05:21:24.810127 9707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:24.832158 9707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:24.835280 9707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:24.835510 9707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:24.842479 9707 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:43427
I20260621 05:21:24.842597 9784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:43427 every 8 connection(s)
I20260621 05:21:24.843022 9707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:21:24.843464 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 9707
I20260621 05:21:24.843581 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:21:24.843782 2176 external_mini_cluster.cc:1428] Setting key 4dc7fe157b89e5dbf330fee513f1f3be
I20260621 05:21:24.843991 9785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:24.848328 9785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9: Bootstrap starting.
I20260621 05:21:24.849156 9785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:24.849478 9785 log.cc:824] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9: Log is configured to *not* fsync() on all Append() calls
I20260621 05:21:24.850183 9785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9: No bootstrap required, opened a new log
Jun 21 05:21:24 dist-test-slave-b357 krb5kdc[9694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM
I20260621 05:21:24.852528 9785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa2a63cb3f94a838d32185a18ab7fe9" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43427 } }
I20260621 05:21:24.852720 9785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:24.852761 9785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aa2a63cb3f94a838d32185a18ab7fe9, State: Initialized, Role: FOLLOWER
I20260621 05:21:24.852921 9785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa2a63cb3f94a838d32185a18ab7fe9" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43427 } }
I20260621 05:21:24.853031 9785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:21:24.853075 9785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:21:24.853124 9785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:24.853832 9785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa2a63cb3f94a838d32185a18ab7fe9" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43427 } }
I20260621 05:21:24.853948 9785 leader_election.cc:302] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8aa2a63cb3f94a838d32185a18ab7fe9; no voters:
I20260621 05:21:24.854099 9785 leader_election.cc:288] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:21:24.854271 9790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:21:24.854346 9785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:21:24.854641 9790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9 [term 1 LEADER]: Becoming Leader. State: Replica: 8aa2a63cb3f94a838d32185a18ab7fe9, State: Running, Role: LEADER
I20260621 05:21:24.854833 9790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa2a63cb3f94a838d32185a18ab7fe9" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43427 } }
I20260621 05:21:24.855773 9791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8aa2a63cb3f94a838d32185a18ab7fe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa2a63cb3f94a838d32185a18ab7fe9" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43427 } } }
I20260621 05:21:24.855891 9791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9 [sys.catalog]: This master's current role is: LEADER
I20260621 05:21:24.856348 9803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:21:24.856292 9792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8aa2a63cb3f94a838d32185a18ab7fe9. Latest consensus state: current_term: 1 leader_uuid: "8aa2a63cb3f94a838d32185a18ab7fe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa2a63cb3f94a838d32185a18ab7fe9" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43427 } } }
I20260621 05:21:24.856465 9792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9 [sys.catalog]: This master's current role is: LEADER
I20260621 05:21:24.856967 9788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:24.844446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35590 (local address 127.2.32.62:43427)
0621 05:21:24.844612 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:24.844622 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:24.844662 (+ 40us) server_negotiation.cc:407] Connection header received
0621 05:21:24.845621 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:24.845629 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:24.845684 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:24.845812 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:24.846798 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:24.847825 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:24.849022 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:24.849505 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:24.849652 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:24.852538 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:24.852542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:24.852543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:21:24.852554 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:24.854582 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:24.855018 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:24.855020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:24.855021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:24.855165 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:24.855525 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:24.855525 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:24.855526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:24.855845 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:24.856035 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:24.856582 (+ 547us) server_negotiation.cc:299] Negotiation successful
0621 05:21:24.856746 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260621 05:21:24.857788 9803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:21:24.859526 9803 catalog_manager.cc:1347] Generated new cluster ID: 5a3fa143e75e450e8591e9304508dec9
I20260621 05:21:24.859586 9803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:21:24.884613 9803 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:21:24.885344 9803 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:21:24.889263 9803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9: Generated new TSK 0
I20260621 05:21:24.889441 9803 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.1.keytab.
Entry for principal oryx/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.1.keytab.
I20260621 05:21:24.925887 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:43427
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.1.keytab
--principal=oryx/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:21:25.009822 9811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:25.010076 9811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:25.010118 9811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:25.011317 9811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:25.011381 9811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:25.011421 9811 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:21:25.011487 9811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:21:25.013056 9811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.1.keytab
--principal=oryx/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.32.62:43427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:25.013288 9811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:25.013460 9811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:25.018627 9819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:25.018752 9811 server_base.cc:1034] running on GCE node
W20260621 05:21:25.018673 9817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:25.018639 9816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:25.019083 9811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:25.019349 9811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:25.020581 9811 hybrid_clock.cc:648] HybridClock initialized: now 1782019285020540 us; error 60 us; skew 500 ppm
Jun 21 05:21:25 dist-test-slave-b357 krb5kdc[9694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019285, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:21:25.023653 9811 init.cc:377] Logged in from keytab as oryx/127.2.32.1@KRBTEST.COM (short username oryx)
I20260621 05:21:25.024142 9811 webserver.cc:460] Webserver started at http://127.2.32.1:37199/ using document root <none> and password file <none>
I20260621 05:21:25.024413 9811 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:25.024461 9811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:25.024582 9811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:21:25.025650 9811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "cfa08fd3741046fd93232e135785615c"
format_stamp: "Formatted at 2026-06-21 05:21:25 on dist-test-slave-b357"
server_key: "f2887f44c03d6c8b9343dee6892bbf68"
server_key_iv: "e69ea6679120acf6f7921e923083572b"
server_key_version: "encryptionkey@0"
I20260621 05:21:25.026026 9811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "cfa08fd3741046fd93232e135785615c"
format_stamp: "Formatted at 2026-06-21 05:21:25 on dist-test-slave-b357"
server_key: "f2887f44c03d6c8b9343dee6892bbf68"
server_key_iv: "e69ea6679120acf6f7921e923083572b"
server_key_version: "encryptionkey@0"
I20260621 05:21:25.027697 9811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:21:25.028884 9826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:25.029145 9811 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:25.029245 9811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "cfa08fd3741046fd93232e135785615c"
format_stamp: "Formatted at 2026-06-21 05:21:25 on dist-test-slave-b357"
server_key: "f2887f44c03d6c8b9343dee6892bbf68"
server_key_iv: "e69ea6679120acf6f7921e923083572b"
server_key_version: "encryptionkey@0"
I20260621 05:21:25.029300 9811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:25.036865 9811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:25.039644 9811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:25.039762 9811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:25.040195 9811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:21:25.040238 9811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:25.040266 9811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:21:25.040324 9811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:25.046974 9811 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:36237
I20260621 05:21:25.047056 9941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:36237 every 8 connection(s)
Jun 21 05:21:25 dist-test-slave-b357 krb5kdc[9694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019285, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM
I20260621 05:21:25.047405 9811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:21:25.050587 9788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.040707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:48177 (local address 127.2.32.62:43427)
0621 05:21:25.040785 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:25.040789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:25.040953 (+ 164us) server_negotiation.cc:407] Connection header received
0621 05:21:25.041483 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:25.041485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:25.041496 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:25.041554 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:25.042365 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.043266 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.044828 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.045155 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.045243 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.047852 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:25.047853 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:25.047853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:21:25.047861 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:25.049213 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:25.049603 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:25.049604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:25.049604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:25.049667 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:25.049924 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:25.049925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:25.049926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:25.050048 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:25.050123 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:25.050398 (+ 275us) server_negotiation.cc:299] Negotiation successful
0621 05:21:25.050495 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 05:21:25.050735 9835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.040677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:43427 (local address 127.2.32.1:48177)
0621 05:21:25.040821 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:25.040866 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:25.041376 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:25.041617 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:25.041621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:25.041977 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:25.042255 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.042260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.043367 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.043368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.044734 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.044738 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.045246 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.045246 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.045346 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.045880 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:25.045891 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:25.047755 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:21:25.049299 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:25.049306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:25.049308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:25.049524 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:25.049735 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:25.049736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:25.049736 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:25.049853 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:25.050116 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:21:25.050122 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:21:25.050258 (+ 136us) client_negotiation.cc:769] Sending connection context
0621 05:21:25.050363 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 05:21:25.050518 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260621 05:21:25.051214 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 9811
I20260621 05:21:25.051338 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:21:25.051537 2176 external_mini_cluster.cc:1428] Setting key d8a2556eea1746a1b969f4cca3019542
I20260621 05:21:25.051769 9942 heartbeater.cc:347] Connected to a master server at 127.2.32.62:43427
I20260621 05:21:25.051898 9942 heartbeater.cc:464] Registering TS with master...
I20260621 05:21:25.052119 9942 heartbeater.cc:511] Master 127.2.32.62:43427 requested a full tablet report, sending...
I20260621 05:21:25.052623 9739 ts_manager.cc:194] Registered new tserver with Master: cfa08fd3741046fd93232e135785615c (127.2.32.1:36237)
I20260621 05:21:25.053228 9739 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:48177
I20260621 05:21:25.061177 9788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.052405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:52299 (local address 127.2.32.62:43427)
0621 05:21:25.052449 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:25.052451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:25.053599 (+ 1148us) server_negotiation.cc:407] Connection header received
0621 05:21:25.053694 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:25.053695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:25.053705 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:25.053734 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:25.054215 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.054894 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.056066 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.056395 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.056496 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.057690 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:25.057691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:25.057692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:21:25.057699 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:25.059802 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:25.060283 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:25.060284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:25.060284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:25.060344 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:25.060553 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:25.060555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:25.060555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:25.060695 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:25.060742 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:25.060967 (+ 225us) server_negotiation.cc:299] Negotiation successful
0621 05:21:25.061046 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:21:25.061498 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.052390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:43427 (local address 127.2.32.1:52299)
0621 05:21:25.053531 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:25.053543 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:25.053626 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:25.053793 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:25.053794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:25.054079 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:25.054146 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.054148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.055054 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.055055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.055970 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.055973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.056522 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.056524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.056615 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.057120 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:25.057126 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:25.057594 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:21:25.059905 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:25.059906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:25.059906 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:25.060188 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:25.060420 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:25.060421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:25.060421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:25.060481 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:25.060797 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:21:25.060797 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:21:25.060845 (+ 48us) client_negotiation.cc:769] Sending connection context
0621 05:21:25.061269 (+ 424us) client_negotiation.cc:240] Negotiation successful
0621 05:21:25.061383 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"thread_start_us":1066,"threads_started":1}
WARNING: no policy specified for oryx/127.2.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.2.keytab.
Entry for principal oryx/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.2.keytab.
I20260621 05:21:25.082437 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:43427
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.2.keytab
--principal=oryx/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:21:25.166785 9951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:25.167181 9951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:25.167232 9951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:25.168476 9951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:25.168524 9951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:25.168545 9951 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:21:25.168561 9951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:21:25.170100 9951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.2.keytab
--principal=oryx/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.32.62:43427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:25.170323 9951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:25.170532 9951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:21:25.175760 9951 server_base.cc:1034] running on GCE node
W20260621 05:21:25.175738 9957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:25.175738 9956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:25.175738 9959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:25.176235 9951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:25.176632 9951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:25.177887 9951 hybrid_clock.cc:648] HybridClock initialized: now 1782019285177869 us; error 44 us; skew 500 ppm
Jun 21 05:21:25 dist-test-slave-b357 krb5kdc[9694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019285, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:21:25.180636 9951 init.cc:377] Logged in from keytab as oryx/127.2.32.2@KRBTEST.COM (short username oryx)
I20260621 05:21:25.181135 9951 webserver.cc:460] Webserver started at http://127.2.32.2:36161/ using document root <none> and password file <none>
I20260621 05:21:25.181367 9951 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:25.181434 9951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:25.181576 9951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:21:25.182598 9951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "0a535c9286b64080b815c5e965783f78"
format_stamp: "Formatted at 2026-06-21 05:21:25 on dist-test-slave-b357"
server_key: "75a625b54692c2210ed154518d85dba5"
server_key_iv: "f3085b31f7c18955aca40a1b323a1950"
server_key_version: "encryptionkey@0"
I20260621 05:21:25.183377 9951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "0a535c9286b64080b815c5e965783f78"
format_stamp: "Formatted at 2026-06-21 05:21:25 on dist-test-slave-b357"
server_key: "75a625b54692c2210ed154518d85dba5"
server_key_iv: "f3085b31f7c18955aca40a1b323a1950"
server_key_version: "encryptionkey@0"
I20260621 05:21:25.185225 9951 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260621 05:21:25.186353 9966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:25.186643 9951 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:25.186771 9951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "0a535c9286b64080b815c5e965783f78"
format_stamp: "Formatted at 2026-06-21 05:21:25 on dist-test-slave-b357"
server_key: "75a625b54692c2210ed154518d85dba5"
server_key_iv: "f3085b31f7c18955aca40a1b323a1950"
server_key_version: "encryptionkey@0"
I20260621 05:21:25.187018 9951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:25.196480 9951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:25.198853 9951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:25.199014 9951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:25.199646 9951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:21:25.199698 9951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:25.199730 9951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:21:25.199748 9951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 05:21:25 dist-test-slave-b357 krb5kdc[9694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019285, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM
I20260621 05:21:25.206609 9951 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:44405
I20260621 05:21:25.206687 10081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:44405 every 8 connection(s)
I20260621 05:21:25.207068 9951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:21:25.207617 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 9951
I20260621 05:21:25.207753 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:21:25.207969 2176 external_mini_cluster.cc:1428] Setting key 5f8c0f9f6cb8e80b24fb7e7ba7aff18f
I20260621 05:21:25.209756 9788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.200172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:35481 (local address 127.2.32.62:43427)
0621 05:21:25.200256 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:25.200259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:25.200392 (+ 133us) server_negotiation.cc:407] Connection header received
0621 05:21:25.200951 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:25.200952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:25.200964 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:25.201011 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:25.201663 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.202572 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.203807 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.204016 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.204099 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.206685 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:25.206686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:25.206686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:21:25.206693 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:25.208368 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:25.208755 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:25.208757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:25.208758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:25.208816 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:25.209122 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:25.209123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:25.209123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:25.209239 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:25.209292 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:25.209634 (+ 342us) server_negotiation.cc:299] Negotiation successful
0621 05:21:25.209695 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 05:21:25.210043 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.200015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:43427 (local address 127.2.32.2:35481)
0621 05:21:25.200250 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:25.200294 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:25.200835 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:25.201092 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:25.201095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:25.201381 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:25.201554 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.201558 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.202670 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.202671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.203689 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.203691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.204119 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.204120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.204233 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.204978 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:25.204986 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:25.206565 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:21:25.208468 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:25.208473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:25.208474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:25.208698 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:25.208900 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:25.208901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:25.208901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:25.209053 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:25.209321 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:21:25.209327 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:21:25.209491 (+ 164us) client_negotiation.cc:769] Sending connection context
0621 05:21:25.209602 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 05:21:25.209772 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":108,"threads_started":1}
I20260621 05:21:25.210556 10082 heartbeater.cc:347] Connected to a master server at 127.2.32.62:43427
I20260621 05:21:25.210637 10082 heartbeater.cc:464] Registering TS with master...
I20260621 05:21:25.210857 10082 heartbeater.cc:511] Master 127.2.32.62:43427 requested a full tablet report, sending...
I20260621 05:21:25.211232 9739 ts_manager.cc:194] Registered new tserver with Master: 0a535c9286b64080b815c5e965783f78 (127.2.32.2:44405)
I20260621 05:21:25.211740 9739 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:35481
WARNING: no policy specified for oryx/127.2.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.32.3@KRBTEST.COM" created.
I20260621 05:21:25.221673 9788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.211580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:44643 (local address 127.2.32.62:43427)
0621 05:21:25.211623 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:25.211625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:25.215471 (+ 3846us) server_negotiation.cc:407] Connection header received
0621 05:21:25.215582 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:25.215583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:25.215597 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:25.215659 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:25.216054 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.216635 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.217404 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.217611 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.217703 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.219011 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:25.219013 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:25.219013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:21:25.219021 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:25.220474 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:25.220838 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:25.220840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:25.220840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:25.220892 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:25.221102 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:25.221103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:25.221103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:25.221250 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:25.221339 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:25.221495 (+ 156us) server_negotiation.cc:299] Negotiation successful
0621 05:21:25.221576 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 05:21:25.221688 10089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.211528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:43427 (local address 127.2.32.2:44643)
0621 05:21:25.215318 (+ 3790us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:25.215335 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:25.215493 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:25.215665 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:25.215666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:25.215898 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:25.215969 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.215971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.216739 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.216740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.217310 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.217312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.217705 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.217706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.217781 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.218478 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:25.218483 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:25.218799 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:21:25.220579 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:25.220581 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:25.220581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:25.220751 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:25.220962 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:25.220963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:25.220964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:25.221029 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:25.221334 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:21:25.221335 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:21:25.221381 (+ 46us) client_negotiation.cc:769] Sending connection context
0621 05:21:25.221470 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 05:21:25.221577 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3749,"thread_start_us":67,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.3.keytab.
Entry for principal oryx/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.3.keytab.
I20260621 05:21:25.234612 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:43427
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.3.keytab
--principal=oryx/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:21:25.316304 10091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:25.316507 10091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:25.316530 10091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:25.317581 10091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:25.317616 10091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:25.317636 10091 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 05:21:25.317653 10091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:21:25.319059 10091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.3.keytab
--principal=oryx/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.32.62:43427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:25.319355 10091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:25.319589 10091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:25.325122 10096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:25.325227 10097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:25.325196 10099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:25.325285 10091 server_base.cc:1034] running on GCE node
I20260621 05:21:25.325482 10091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:25.325757 10091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:25.326946 10091 hybrid_clock.cc:648] HybridClock initialized: now 1782019285326933 us; error 29 us; skew 500 ppm
Jun 21 05:21:25 dist-test-slave-b357 krb5kdc[9694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019285, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:21:25.329821 10091 init.cc:377] Logged in from keytab as oryx/127.2.32.3@KRBTEST.COM (short username oryx)
I20260621 05:21:25.330315 10091 webserver.cc:460] Webserver started at http://127.2.32.3:46713/ using document root <none> and password file <none>
I20260621 05:21:25.330564 10091 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:25.330632 10091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:25.330793 10091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:21:25.332008 10091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "889e832a1c314be69ec5921b3abc3cd5"
format_stamp: "Formatted at 2026-06-21 05:21:25 on dist-test-slave-b357"
server_key: "1865c0c63d284119b85d28223156a3a3"
server_key_iv: "41a7826568bc407fd218759dc33ba56d"
server_key_version: "encryptionkey@0"
I20260621 05:21:25.332391 10091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "889e832a1c314be69ec5921b3abc3cd5"
format_stamp: "Formatted at 2026-06-21 05:21:25 on dist-test-slave-b357"
server_key: "1865c0c63d284119b85d28223156a3a3"
server_key_iv: "41a7826568bc407fd218759dc33ba56d"
server_key_version: "encryptionkey@0"
I20260621 05:21:25.333935 10091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:21:25.334914 10106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:25.335254 10091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:25.335372 10091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "889e832a1c314be69ec5921b3abc3cd5"
format_stamp: "Formatted at 2026-06-21 05:21:25 on dist-test-slave-b357"
server_key: "1865c0c63d284119b85d28223156a3a3"
server_key_iv: "41a7826568bc407fd218759dc33ba56d"
server_key_version: "encryptionkey@0"
I20260621 05:21:25.335457 10091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:25.346946 10091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:25.349812 10091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:25.350024 10091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:25.350652 10091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:21:25.350762 10091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:25.350826 10091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:21:25.350849 10091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 05:21:25 dist-test-slave-b357 krb5kdc[9694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019285, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM
I20260621 05:21:25.360262 10091 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:37013
I20260621 05:21:25.360363 10221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:37013 every 8 connection(s)
I20260621 05:21:25.360832 10091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:21:25.361912 9788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.351158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:52145 (local address 127.2.32.62:43427)
0621 05:21:25.351231 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:25.351235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:25.351677 (+ 442us) server_negotiation.cc:407] Connection header received
0621 05:21:25.352713 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:25.352716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:25.352730 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:25.352809 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:25.353986 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.354537 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.355896 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.356107 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.356174 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.358973 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:25.358974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:25.358975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:21:25.358982 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:25.360603 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:25.361016 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:25.361017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:25.361018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:25.361088 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:25.361371 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:25.361372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:25.361373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:25.361510 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:25.361564 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:25.361784 (+ 220us) server_negotiation.cc:299] Negotiation successful
0621 05:21:25.361850 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 05:21:25.362107 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.351190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:43427 (local address 127.2.32.3:52145)
0621 05:21:25.351453 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:25.351510 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:25.352506 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:25.352941 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:25.352946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:25.353467 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:25.353863 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.353869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.354658 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.354660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.355769 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.355772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.356263 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.356264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.356418 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.356931 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:25.356941 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:25.358863 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:21:25.360696 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:25.360702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:25.360704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:25.360948 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:25.361169 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:25.361170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:25.361170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:25.361307 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:25.361558 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:21:25.361560 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:21:25.361664 (+ 104us) client_negotiation.cc:769] Sending connection context
0621 05:21:25.361750 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 05:21:25.361873 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":128,"threads_started":1}
I20260621 05:21:25.362692 10222 heartbeater.cc:347] Connected to a master server at 127.2.32.62:43427
I20260621 05:21:25.362777 10222 heartbeater.cc:464] Registering TS with master...
I20260621 05:21:25.362967 10222 heartbeater.cc:511] Master 127.2.32.62:43427 requested a full tablet report, sending...
I20260621 05:21:25.363713 9738 ts_manager.cc:194] Registered new tserver with Master: 889e832a1c314be69ec5921b3abc3cd5 (127.2.32.3:37013)
I20260621 05:21:25.364425 9738 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:52145
I20260621 05:21:25.369792 9788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.363698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:59641 (local address 127.2.32.62:43427)
0621 05:21:25.363748 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:25.363751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:25.363911 (+ 160us) server_negotiation.cc:407] Connection header received
0621 05:21:25.364006 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:25.364007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:25.364015 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:25.364067 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:25.364553 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.365102 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.366026 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.366243 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.366317 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.367395 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:25.367396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:25.367396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:21:25.367402 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:25.368769 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:25.369141 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:25.369142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:25.369142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:25.369206 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:25.369384 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:25.369385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:25.369385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:25.369486 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:25.369539 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:25.369659 (+ 120us) server_negotiation.cc:299] Negotiation successful
0621 05:21:25.369725 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260621 05:21:25.369807 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.363648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:43427 (local address 127.2.32.3:59641)
0621 05:21:25.363815 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:25.363832 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:25.363933 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:25.364078 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:25.364079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:25.364336 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:25.364429 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.364431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.365218 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.365219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.365904 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.365907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.366369 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.366370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.366430 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.366918 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:25.366925 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:25.367292 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:21:25.368887 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:25.368888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:25.368888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:25.369064 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:25.369274 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:25.369274 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:25.369274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:25.369333 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:25.369546 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:21:25.369547 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:21:25.369580 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 05:21:25.369638 (+ 58us) client_negotiation.cc:240] Negotiation successful
0621 05:21:25.369734 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260621 05:21:25.370151 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 10091
I20260621 05:21:25.370255 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:21:25.370456 2176 external_mini_cluster.cc:1428] Setting key 324feaec17026b33927702081b7c8989
I20260621 05:21:25.371371 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 05:21:25 dist-test-slave-b357 krb5kdc[9694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019285, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 05:21:25 dist-test-slave-b357 krb5kdc[9694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:25 dist-test-slave-b357 krb5kdc[9694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 05:21:25.396157 9788 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:25.389683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35600 (local address 127.2.32.62:43427)
0621 05:21:25.389754 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:25.389756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:25.389767 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:21:25.389843 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:25.389844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:25.389862 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:25.389943 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:25.390676 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.391259 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.392125 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.392316 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.392408 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.396057 (+ 3649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35600: BlockingRecv error: recv got EOF from 127.0.0.1:35600 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 21 05:21:25 dist-test-slave-b357 krb5kdc[9694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019285, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM
I20260621 05:21:25.404740 9788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.397065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35610 (local address 127.2.32.62:43427)
0621 05:21:25.397128 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:25.397130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:25.397139 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:21:25.397193 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:25.397194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:25.397203 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:25.397257 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:25.397719 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.398163 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.398828 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.399015 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.399100 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.401199 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:25.401200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:25.401201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:21:25.401207 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:25.402959 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:25.403513 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:25.403515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:25.403515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:25.403590 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:25.403843 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:25.403844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:25.403845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:25.404148 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:25.404309 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:25.404495 (+ 186us) server_negotiation.cc:299] Negotiation successful
0621 05:21:25.404597 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 05:21:25.406822 9738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:21:25.408730 9738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:21:25.416309 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.412031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:44405 (local address 127.0.0.1:48748)
0621 05:21:25.412179 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:25.412225 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:25.412305 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:25.412851 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:25.412854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:25.412857 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:21:25.412926 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.412928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.414058 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.414059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.415202 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.415205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.415984 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.415985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.416080 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.416093 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 05:21:25.416165 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 05:21:25.416227 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":120,"threads_started":1}
I20260621 05:21:25.416646 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.412301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48748 (local address 127.2.32.2:44405)
0621 05:21:25.412692 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:25.412697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:25.412706 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:21:25.412747 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:25.412749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:25.412766 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:25.412846 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:21:25.413010 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.413982 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.415417 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.415877 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.415980 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.416057 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:25.416501 (+ 444us) server_negotiation.cc:299] Negotiation successful
0621 05:21:25.416541 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":147,"threads_started":1}
I20260621 05:21:25.417346 10016 tablet_service.cc:1467] Processing CreateTablet for tablet 69817b9b65114afeaf4accbee2090736 (DEFAULT_TABLE table=test-table [id=3650a1750aef4dcc8a6d9d49eae1c324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:21:25.417583 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.412291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:37013 (local address 127.0.0.1:33026)
0621 05:21:25.412796 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:25.412809 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:25.412875 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:25.413357 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:25.413358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:25.413361 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:21:25.413421 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.413423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.414839 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.414840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.416080 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.416082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.417229 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.417230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.417320 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.417325 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:21:25.417433 (+ 108us) client_negotiation.cc:240] Negotiation successful
0621 05:21:25.417491 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":100,"threads_started":1}
I20260621 05:21:25.417635 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.412571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33026 (local address 127.2.32.3:37013)
0621 05:21:25.413118 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:25.413125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:25.413135 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:21:25.413183 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:25.413185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:25.413205 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:25.413356 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:21:25.413497 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.414743 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.416172 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.417096 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.417191 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.417265 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:25.417450 (+ 185us) server_negotiation.cc:299] Negotiation successful
0621 05:21:25.417508 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":120,"threads_started":1}
I20260621 05:21:25.417732 10016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69817b9b65114afeaf4accbee2090736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:25.418211 10156 tablet_service.cc:1467] Processing CreateTablet for tablet 69817b9b65114afeaf4accbee2090736 (DEFAULT_TABLE table=test-table [id=3650a1750aef4dcc8a6d9d49eae1c324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:21:25.418462 10156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69817b9b65114afeaf4accbee2090736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:25.421420 10250 tablet_bootstrap.cc:492] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5: Bootstrap starting.
I20260621 05:21:25.421528 10251 tablet_bootstrap.cc:492] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78: Bootstrap starting.
I20260621 05:21:25.422073 10251 tablet_bootstrap.cc:654] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:25.422073 10250 tablet_bootstrap.cc:654] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:25.422284 10247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.412034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:36237 (local address 127.0.0.1:41638)
0621 05:21:25.412438 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:25.412450 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:25.412539 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:25.412948 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:25.412949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:25.412952 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:21:25.412996 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.412997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.414620 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.414622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.416041 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.416044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.421956 (+ 5912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.421958 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.422049 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.422055 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:21:25.422131 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 05:21:25.422188 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":84,"threads_started":1}
I20260621 05:21:25.422354 10246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.412142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41638 (local address 127.2.32.1:36237)
0621 05:21:25.412337 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:25.412342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:25.412511 (+ 169us) server_negotiation.cc:407] Connection header received
0621 05:21:25.412754 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:25.412760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:25.412784 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:25.412944 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:21:25.413114 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.414493 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.416146 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.416943 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.417120 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.417262 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:25.422181 (+ 4919us) server_negotiation.cc:299] Negotiation successful
0621 05:21:25.422245 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":92,"threads_started":1}
I20260621 05:21:25.422422 10250 log.cc:824] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5: Log is configured to *not* fsync() on all Append() calls
I20260621 05:21:25.423177 10250 tablet_bootstrap.cc:492] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5: No bootstrap required, opened a new log
I20260621 05:21:25.423070 9876 tablet_service.cc:1467] Processing CreateTablet for tablet 69817b9b65114afeaf4accbee2090736 (DEFAULT_TABLE table=test-table [id=3650a1750aef4dcc8a6d9d49eae1c324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:21:25.423230 10251 log.cc:824] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78: Log is configured to *not* fsync() on all Append() calls
I20260621 05:21:25.423265 10250 ts_tablet_manager.cc:1397] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:21:25.423470 9876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69817b9b65114afeaf4accbee2090736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:25.423988 10251 tablet_bootstrap.cc:492] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78: No bootstrap required, opened a new log
I20260621 05:21:25.424115 10251 ts_tablet_manager.cc:1397] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:21:25.425009 10250 raft_consensus.cc:348] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } }
I20260621 05:21:25.425163 10250 raft_consensus.cc:374] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:25.425204 10250 raft_consensus.cc:729] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889e832a1c314be69ec5921b3abc3cd5, State: Initialized, Role: FOLLOWER
I20260621 05:21:25.425328 10250 consensus_queue.cc:260] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } }
I20260621 05:21:25.425602 10250 ts_tablet_manager.cc:1428] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:21:25.425635 10222 heartbeater.cc:503] Master 127.2.32.62:43427 was elected leader, sending a full tablet report...
I20260621 05:21:25.426242 10251 raft_consensus.cc:348] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } }
I20260621 05:21:25.426445 10251 raft_consensus.cc:374] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:25.426482 10251 raft_consensus.cc:729] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a535c9286b64080b815c5e965783f78, State: Initialized, Role: FOLLOWER
I20260621 05:21:25.426602 10251 consensus_queue.cc:260] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } }
I20260621 05:21:25.426857 10251 ts_tablet_manager.cc:1428] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:21:25.426983 10082 heartbeater.cc:503] Master 127.2.32.62:43427 was elected leader, sending a full tablet report...
I20260621 05:21:25.427874 10256 tablet_bootstrap.cc:492] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c: Bootstrap starting.
I20260621 05:21:25.428695 10256 tablet_bootstrap.cc:654] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:25.429055 10256 log.cc:824] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c: Log is configured to *not* fsync() on all Append() calls
I20260621 05:21:25.429941 10256 tablet_bootstrap.cc:492] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c: No bootstrap required, opened a new log
I20260621 05:21:25.430047 10256 ts_tablet_manager.cc:1397] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:21:25.431658 10256 raft_consensus.cc:348] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } }
I20260621 05:21:25.431846 10256 raft_consensus.cc:374] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:25.431918 10256 raft_consensus.cc:729] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfa08fd3741046fd93232e135785615c, State: Initialized, Role: FOLLOWER
I20260621 05:21:25.432057 10256 consensus_queue.cc:260] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } }
I20260621 05:21:25.432332 10256 ts_tablet_manager.cc:1428] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260621 05:21:25.432466 9942 heartbeater.cc:503] Master 127.2.32.62:43427 was elected leader, sending a full tablet report...
W20260621 05:21:25.457674 10083 tablet.cc:2369] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:21:25.494335 10255 raft_consensus.cc:482] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:21:25.494525 10255 raft_consensus.cc:504] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } }
I20260621 05:21:25.494918 10255 leader_election.cc:288] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfa08fd3741046fd93232e135785615c (127.2.32.1:36237), 889e832a1c314be69ec5921b3abc3cd5 (127.2.32.3:37013)
I20260621 05:21:25.498555 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.495553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:54443 (local address 127.2.32.3:37013)
0621 05:21:25.495629 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:25.495631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:25.495754 (+ 123us) server_negotiation.cc:407] Connection header received
0621 05:21:25.495812 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:25.495813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:25.495824 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:25.495880 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:21:25.496124 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.496879 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.497826 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.498184 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.498258 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.498271 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:25.498437 (+ 166us) server_negotiation.cc:299] Negotiation successful
0621 05:21:25.498473 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 05:21:25.498582 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.495380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:37013 (local address 127.2.32.2:54443)
0621 05:21:25.495620 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:25.495635 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:25.495744 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:25.495983 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:25.495983 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:25.495985 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:21:25.496041 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.496043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.496946 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.496948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.497721 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.497723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.498280 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.498281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.498360 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.498364 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:21:25.498432 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 05:21:25.498497 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":23,"thread_start_us":152,"threads_started":1}
I20260621 05:21:25.498677 10246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.495437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:36669 (local address 127.2.32.1:36237)
0621 05:21:25.495526 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:25.495529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:25.495539 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:21:25.495567 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:25.495568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:25.495583 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:25.495646 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:21:25.495883 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.496625 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.497870 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.498288 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.498386 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.498396 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:25.498573 (+ 177us) server_negotiation.cc:299] Negotiation successful
0621 05:21:25.498611 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 05:21:25.498701 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.495233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:36237 (local address 127.2.32.2:36669)
0621 05:21:25.495300 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:25.495313 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:25.495363 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:25.495685 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:25.495687 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:25.495689 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:21:25.495787 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.495790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.496752 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.496753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.497743 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.497745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.498384 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.498385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.498480 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.498483 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:21:25.498554 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 05:21:25.498596 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260621 05:21:25.499200 9896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69817b9b65114afeaf4accbee2090736" candidate_uuid: "0a535c9286b64080b815c5e965783f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfa08fd3741046fd93232e135785615c" is_pre_election: true
I20260621 05:21:25.499181 10175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69817b9b65114afeaf4accbee2090736" candidate_uuid: "0a535c9286b64080b815c5e965783f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889e832a1c314be69ec5921b3abc3cd5" is_pre_election: true
I20260621 05:21:25.499351 10175 raft_consensus.cc:2454] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a535c9286b64080b815c5e965783f78 in term 0.
I20260621 05:21:25.499351 9896 raft_consensus.cc:2454] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a535c9286b64080b815c5e965783f78 in term 0.
I20260621 05:21:25.499568 9970 leader_election.cc:302] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a535c9286b64080b815c5e965783f78, cfa08fd3741046fd93232e135785615c; no voters:
I20260621 05:21:25.499693 10255 raft_consensus.cc:2781] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:21:25.499747 10255 raft_consensus.cc:482] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:21:25.499775 10255 raft_consensus.cc:3037] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:25.500710 10255 raft_consensus.cc:504] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } }
I20260621 05:21:25.500852 10255 leader_election.cc:288] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [CANDIDATE]: Term 1 election: Requested vote from peers cfa08fd3741046fd93232e135785615c (127.2.32.1:36237), 889e832a1c314be69ec5921b3abc3cd5 (127.2.32.3:37013)
I20260621 05:21:25.501010 10175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69817b9b65114afeaf4accbee2090736" candidate_uuid: "0a535c9286b64080b815c5e965783f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889e832a1c314be69ec5921b3abc3cd5"
I20260621 05:21:25.501034 9896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69817b9b65114afeaf4accbee2090736" candidate_uuid: "0a535c9286b64080b815c5e965783f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfa08fd3741046fd93232e135785615c"
I20260621 05:21:25.501094 9896 raft_consensus.cc:3037] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:25.501088 10175 raft_consensus.cc:3037] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:25.501904 9896 raft_consensus.cc:2454] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a535c9286b64080b815c5e965783f78 in term 1.
I20260621 05:21:25.501914 10175 raft_consensus.cc:2454] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a535c9286b64080b815c5e965783f78 in term 1.
I20260621 05:21:25.502169 9970 leader_election.cc:302] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a535c9286b64080b815c5e965783f78, cfa08fd3741046fd93232e135785615c; no voters:
I20260621 05:21:25.502270 10255 raft_consensus.cc:2781] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:21:25.502429 10255 raft_consensus.cc:686] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [term 1 LEADER]: Becoming Leader. State: Replica: 0a535c9286b64080b815c5e965783f78, State: Running, Role: LEADER
I20260621 05:21:25.502521 10255 consensus_queue.cc:237] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } }
I20260621 05:21:25.503361 9738 catalog_manager.cc:5495] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a535c9286b64080b815c5e965783f78 (127.2.32.2). New cstate: current_term: 1 leader_uuid: "0a535c9286b64080b815c5e965783f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } health_report { overall_health: HEALTHY } } }
I20260621 05:21:25.510437 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.507723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48754 (local address 127.2.32.2:44405)
0621 05:21:25.507809 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:25.507811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:25.507824 (+ 13us) server_negotiation.cc:407] Connection header received
0621 05:21:25.507868 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:25.507869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:25.507896 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:25.507975 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:21:25.508159 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.508667 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.509494 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.509692 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.509768 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.510060 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:21:25.510202 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:21:25.510278 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:25.510357 (+ 79us) server_negotiation.cc:299] Negotiation successful
0621 05:21:25.510386 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 05:21:25.512832 9896 raft_consensus.cc:1261] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c [term 1 FOLLOWER]: Refusing update from remote peer 0a535c9286b64080b815c5e965783f78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:21:25.512832 10175 raft_consensus.cc:1261] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5 [term 1 FOLLOWER]: Refusing update from remote peer 0a535c9286b64080b815c5e965783f78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:21:25.513232 10260 consensus_queue.cc:1035] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:21:25.513494 10255 consensus_queue.cc:1035] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:21:25.516605 10265 mvcc.cc:205] Tried to move back new op lower bound from 7299150993457209344 to 7299150993418874880. Current Snapshot: MvccSnapshot[applied={T|T < 7299150993457209344}]
I20260621 05:21:25.519805 9739 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:21:25.519917 9739 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:21:25.520780 9739 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9: Sending DeleteTablet for 3 replicas of tablet 69817b9b65114afeaf4accbee2090736
I20260621 05:21:25.521250 10016 tablet_service.cc:1514] Processing DeleteTablet for tablet 69817b9b65114afeaf4accbee2090736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:21:25 UTC) from {username='oryx', principal='oryx/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:48748
I20260621 05:21:25.521258 10156 tablet_service.cc:1514] Processing DeleteTablet for tablet 69817b9b65114afeaf4accbee2090736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:21:25 UTC) from {username='oryx', principal='oryx/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:33026
I20260621 05:21:25.521504 10272 tablet_replica.cc:330] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78: stopping tablet replica
I20260621 05:21:25.521621 10272 raft_consensus.cc:2229] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:21:25.521607 9876 tablet_service.cc:1514] Processing DeleteTablet for tablet 69817b9b65114afeaf4accbee2090736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:21:25 UTC) from {username='oryx', principal='oryx/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:41638
I20260621 05:21:25.521798 10272 raft_consensus.cc:2258] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:21:25.521932 10274 tablet_replica.cc:330] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c: stopping tablet replica
I20260621 05:21:25.522074 10274 raft_consensus.cc:2229] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:21:25.522234 10274 raft_consensus.cc:2258] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:21:25.522159 9739 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:21:25.522331 9739 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:21:25.522481 10272 ts_tablet_manager.cc:1905] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:21:25.523279 10274 ts_tablet_manager.cc:1905] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:21:25.524206 10016 tablet_service.cc:1467] Processing CreateTablet for tablet f43d5e6fcb174aea83dc442f2b9b5e5f (DEFAULT_TABLE table=test-table [id=5da8ff0591b24f84a5572034d29dd2ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:21:25.524354 9876 tablet_service.cc:1467] Processing CreateTablet for tablet f43d5e6fcb174aea83dc442f2b9b5e5f (DEFAULT_TABLE table=test-table [id=5da8ff0591b24f84a5572034d29dd2ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:21:25.524461 9876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f43d5e6fcb174aea83dc442f2b9b5e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:25.525874 10274 ts_tablet_manager.cc:1918] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:21:25.525928 10256 tablet_bootstrap.cc:492] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c: Bootstrap starting.
I20260621 05:21:25.526125 10156 tablet_service.cc:1467] Processing CreateTablet for tablet f43d5e6fcb174aea83dc442f2b9b5e5f (DEFAULT_TABLE table=test-table [id=5da8ff0591b24f84a5572034d29dd2ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:21:25.526276 10156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f43d5e6fcb174aea83dc442f2b9b5e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:25.526510 10256 tablet_bootstrap.cc:654] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:25.526007 10274 log.cc:1196] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-0/wal/wals/69817b9b65114afeaf4accbee2090736
I20260621 05:21:25.527096 10274 ts_tablet_manager.cc:1939] T 69817b9b65114afeaf4accbee2090736 P cfa08fd3741046fd93232e135785615c: Deleting consensus metadata
I20260621 05:21:25.527216 10256 tablet_bootstrap.cc:492] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c: No bootstrap required, opened a new log
I20260621 05:21:25.527259 10256 ts_tablet_manager.cc:1397] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260621 05:21:25.527407 10256 raft_consensus.cc:348] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:25.527477 10256 raft_consensus.cc:374] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:25.527499 10256 raft_consensus.cc:729] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfa08fd3741046fd93232e135785615c, State: Initialized, Role: FOLLOWER
I20260621 05:21:25.527549 10256 consensus_queue.cc:260] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:25.527632 10256 ts_tablet_manager.cc:1428] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:25.527781 10273 tablet_replica.cc:330] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5: stopping tablet replica
I20260621 05:21:25.527884 10273 raft_consensus.cc:2229] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:21:25.527978 10250 tablet_bootstrap.cc:492] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5: Bootstrap starting.
I20260621 05:21:25.527990 10273 raft_consensus.cc:2258] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:21:25.528221 9726 catalog_manager.cc:4841] TS cfa08fd3741046fd93232e135785615c (127.2.32.1:36237): tablet 69817b9b65114afeaf4accbee2090736 (table test-table [id=3650a1750aef4dcc8a6d9d49eae1c324]) successfully deleted
I20260621 05:21:25.528424 10250 tablet_bootstrap.cc:654] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:25.528817 10273 ts_tablet_manager.cc:1905] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:21:25.529326 10016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f43d5e6fcb174aea83dc442f2b9b5e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:25.530189 10272 ts_tablet_manager.cc:1918] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:21:25.530222 10273 ts_tablet_manager.cc:1918] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:21:25.530247 10272 log.cc:1196] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/69817b9b65114afeaf4accbee2090736
I20260621 05:21:25.530282 10273 log.cc:1196] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/69817b9b65114afeaf4accbee2090736
I20260621 05:21:25.530490 10272 ts_tablet_manager.cc:1939] T 69817b9b65114afeaf4accbee2090736 P 0a535c9286b64080b815c5e965783f78: Deleting consensus metadata
I20260621 05:21:25.530529 10273 ts_tablet_manager.cc:1939] T 69817b9b65114afeaf4accbee2090736 P 889e832a1c314be69ec5921b3abc3cd5: Deleting consensus metadata
I20260621 05:21:25.530747 10251 tablet_bootstrap.cc:492] T f43d5e6fcb174aea83dc442f2b9b5e5f P 0a535c9286b64080b815c5e965783f78: Bootstrap starting.
I20260621 05:21:25.530918 9726 catalog_manager.cc:4841] TS 889e832a1c314be69ec5921b3abc3cd5 (127.2.32.3:37013): tablet 69817b9b65114afeaf4accbee2090736 (table test-table [id=3650a1750aef4dcc8a6d9d49eae1c324]) successfully deleted
I20260621 05:21:25.531172 10251 tablet_bootstrap.cc:654] T f43d5e6fcb174aea83dc442f2b9b5e5f P 0a535c9286b64080b815c5e965783f78: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:25.531167 9724 catalog_manager.cc:4841] TS 0a535c9286b64080b815c5e965783f78 (127.2.32.2:44405): tablet 69817b9b65114afeaf4accbee2090736 (table test-table [id=3650a1750aef4dcc8a6d9d49eae1c324]) successfully deleted
I20260621 05:21:25.531539 10250 tablet_bootstrap.cc:492] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5: No bootstrap required, opened a new log
I20260621 05:21:25.531596 10250 ts_tablet_manager.cc:1397] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260621 05:21:25.531749 10251 tablet_bootstrap.cc:492] T f43d5e6fcb174aea83dc442f2b9b5e5f P 0a535c9286b64080b815c5e965783f78: No bootstrap required, opened a new log
I20260621 05:21:25.531759 10250 raft_consensus.cc:348] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:25.531811 10251 ts_tablet_manager.cc:1397] T f43d5e6fcb174aea83dc442f2b9b5e5f P 0a535c9286b64080b815c5e965783f78: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260621 05:21:25.531809 10250 raft_consensus.cc:374] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:25.531852 10250 raft_consensus.cc:729] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889e832a1c314be69ec5921b3abc3cd5, State: Initialized, Role: FOLLOWER
I20260621 05:21:25.531908 10250 consensus_queue.cc:260] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:25.531965 10251 raft_consensus.cc:348] T f43d5e6fcb174aea83dc442f2b9b5e5f P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:25.532022 10250 ts_tablet_manager.cc:1428] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:21:25.532037 10251 raft_consensus.cc:374] T f43d5e6fcb174aea83dc442f2b9b5e5f P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:25.532061 10251 raft_consensus.cc:729] T f43d5e6fcb174aea83dc442f2b9b5e5f P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a535c9286b64080b815c5e965783f78, State: Initialized, Role: FOLLOWER
I20260621 05:21:25.532104 10251 consensus_queue.cc:260] T f43d5e6fcb174aea83dc442f2b9b5e5f P 0a535c9286b64080b815c5e965783f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:25.532200 10251 ts_tablet_manager.cc:1428] T f43d5e6fcb174aea83dc442f2b9b5e5f P 0a535c9286b64080b815c5e965783f78: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260621 05:21:25.548271 9943 tablet.cc:2369] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:21:25.611773 10223 tablet.cc:2369] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:21:25.808256 10254 raft_consensus.cc:482] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:21:25.808429 10254 raft_consensus.cc:504] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:25.808944 10254 leader_election.cc:288] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfa08fd3741046fd93232e135785615c (127.2.32.1:36237), 0a535c9286b64080b815c5e965783f78 (127.2.32.2:44405)
I20260621 05:21:25.812255 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.809336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:58615 (local address 127.2.32.2:44405)
0621 05:21:25.809407 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:25.809409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:25.809614 (+ 205us) server_negotiation.cc:407] Connection header received
0621 05:21:25.809688 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:25.809689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:25.809698 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:25.809751 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:21:25.809910 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.810376 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.811430 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.811906 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.811967 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.811978 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:25.812126 (+ 148us) server_negotiation.cc:299] Negotiation successful
0621 05:21:25.812163 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 05:21:25.812336 10246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.809368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:44925 (local address 127.2.32.1:36237)
0621 05:21:25.809449 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:25.809452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:25.809462 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:21:25.809491 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:25.809492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:25.809509 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:25.809581 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:21:25.809749 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.810244 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.811449 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.812010 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.812084 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.812093 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:25.812247 (+ 154us) server_negotiation.cc:299] Negotiation successful
0621 05:21:25.812277 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 05:21:25.812336 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.809223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:36237 (local address 127.2.32.3:44925)
0621 05:21:25.809270 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:25.809282 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:25.809333 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:25.809586 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:25.809587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:25.809588 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:21:25.809676 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.809678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.810332 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.810333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.811364 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.811366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.812085 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.812086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.812161 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.812165 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:21:25.812222 (+ 57us) client_negotiation.cc:240] Negotiation successful
0621 05:21:25.812261 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260621 05:21:25.812685 10278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.809223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:44405 (local address 127.2.32.3:58615)
0621 05:21:25.809513 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:25.809530 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:25.809627 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:25.809757 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:25.809757 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:25.809759 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:21:25.809841 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.809842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.810460 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.810461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.811332 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.811335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.811966 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.811967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.812040 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.812043 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:21:25.812550 (+ 507us) client_negotiation.cc:240] Negotiation successful
0621 05:21:25.812589 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":160,"thread_start_us":73,"threads_started":1}
I20260621 05:21:25.812817 9896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f43d5e6fcb174aea83dc442f2b9b5e5f" candidate_uuid: "889e832a1c314be69ec5921b3abc3cd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfa08fd3741046fd93232e135785615c" is_pre_election: true
I20260621 05:21:25.812924 9896 raft_consensus.cc:2454] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 889e832a1c314be69ec5921b3abc3cd5 in term 0.
I20260621 05:21:25.813117 10110 leader_election.cc:302] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889e832a1c314be69ec5921b3abc3cd5, cfa08fd3741046fd93232e135785615c; no voters:
I20260621 05:21:25.813190 10036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f43d5e6fcb174aea83dc442f2b9b5e5f" candidate_uuid: "889e832a1c314be69ec5921b3abc3cd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a535c9286b64080b815c5e965783f78" is_pre_election: true
I20260621 05:21:25.813271 10254 raft_consensus.cc:2781] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:21:25.813304 10254 raft_consensus.cc:482] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:21:25.813333 10254 raft_consensus.cc:3037] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:25.813320 10036 raft_consensus.cc:2454] T f43d5e6fcb174aea83dc442f2b9b5e5f P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 889e832a1c314be69ec5921b3abc3cd5 in term 0.
I20260621 05:21:25.814062 10254 raft_consensus.cc:504] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:25.814205 10254 leader_election.cc:288] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [CANDIDATE]: Term 1 election: Requested vote from peers cfa08fd3741046fd93232e135785615c (127.2.32.1:36237), 0a535c9286b64080b815c5e965783f78 (127.2.32.2:44405)
I20260621 05:21:25.814500 9896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f43d5e6fcb174aea83dc442f2b9b5e5f" candidate_uuid: "889e832a1c314be69ec5921b3abc3cd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfa08fd3741046fd93232e135785615c"
I20260621 05:21:25.814456 10036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f43d5e6fcb174aea83dc442f2b9b5e5f" candidate_uuid: "889e832a1c314be69ec5921b3abc3cd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a535c9286b64080b815c5e965783f78"
I20260621 05:21:25.814584 9896 raft_consensus.cc:3037] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:25.814587 10036 raft_consensus.cc:3037] T f43d5e6fcb174aea83dc442f2b9b5e5f P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:25.815255 9896 raft_consensus.cc:2454] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889e832a1c314be69ec5921b3abc3cd5 in term 1.
I20260621 05:21:25.815475 10110 leader_election.cc:302] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889e832a1c314be69ec5921b3abc3cd5, cfa08fd3741046fd93232e135785615c; no voters:
I20260621 05:21:25.815621 10254 raft_consensus.cc:2781] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:21:25.815830 10036 raft_consensus.cc:2454] T f43d5e6fcb174aea83dc442f2b9b5e5f P 0a535c9286b64080b815c5e965783f78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889e832a1c314be69ec5921b3abc3cd5 in term 1.
I20260621 05:21:25.815845 10254 raft_consensus.cc:686] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [term 1 LEADER]: Becoming Leader. State: Replica: 889e832a1c314be69ec5921b3abc3cd5, State: Running, Role: LEADER
I20260621 05:21:25.815937 10254 consensus_queue.cc:237] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:25.816757 9739 catalog_manager.cc:5495] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 889e832a1c314be69ec5921b3abc3cd5 (127.2.32.3). New cstate: current_term: 1 leader_uuid: "889e832a1c314be69ec5921b3abc3cd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } health_report { overall_health: HEALTHY } } }
Jun 21 05:21:25 dist-test-slave-b357 krb5kdc[9694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019284, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.62@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM
I20260621 05:21:25.924502 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.916188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:43427 (local address 127.0.0.1:35614)
0621 05:21:25.916505 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:25.916517 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:25.916617 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:25.916810 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:25.916811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:25.917241 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:25.917292 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.917294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.918242 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.918243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.918876 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.918878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.919392 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.919393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.919443 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.920084 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:25.920090 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:25.921546 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:21:25.923280 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:25.923287 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:25.923289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:25.923507 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:25.923794 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:25.923795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:25.923796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:25.923859 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:25.924190 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:21:25.924196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:21:25.924246 (+ 50us) client_negotiation.cc:769] Sending connection context
0621 05:21:25.924326 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 05:21:25.924427 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":142,"threads_started":1}
I20260621 05:21:25.924585 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.916302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.2.32.62:43427)
0621 05:21:25.916494 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:25.916497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:25.916596 (+ 99us) server_negotiation.cc:407] Connection header received
0621 05:21:25.916695 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:25.916695 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:25.916708 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:25.916805 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:25.917385 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.918072 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.919032 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.919299 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.919392 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.921687 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:25.921689 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:25.921689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:21:25.921696 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:25.923091 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:25.923612 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:25.923614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:25.923614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:25.923684 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:25.923948 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:25.923949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:25.923950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:25.924094 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:25.924169 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:25.924453 (+ 284us) server_negotiation.cc:299] Negotiation successful
0621 05:21:25.924522 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":73,"threads_started":1}
I20260621 05:21:25.927242 9739 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:35614:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 05:21:25.927513 9739 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:21:25.929520 9876 tablet_service.cc:1467] Processing CreateTablet for tablet e6ee8c7971e041008b60f1eb07512db6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=35b298888a574a689936fc391493616c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:21:25.929713 9876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ee8c7971e041008b60f1eb07512db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:25.929698 10156 tablet_service.cc:1467] Processing CreateTablet for tablet e6ee8c7971e041008b60f1eb07512db6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=35b298888a574a689936fc391493616c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:21:25.930061 10156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ee8c7971e041008b60f1eb07512db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:25.931032 10256 tablet_bootstrap.cc:492] T e6ee8c7971e041008b60f1eb07512db6 P cfa08fd3741046fd93232e135785615c: Bootstrap starting.
I20260621 05:21:25.931658 10256 tablet_bootstrap.cc:654] T e6ee8c7971e041008b60f1eb07512db6 P cfa08fd3741046fd93232e135785615c: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:25.932372 10016 tablet_service.cc:1467] Processing CreateTablet for tablet e6ee8c7971e041008b60f1eb07512db6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=35b298888a574a689936fc391493616c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 05:21:25.932488 10250 tablet_bootstrap.cc:492] T e6ee8c7971e041008b60f1eb07512db6 P 889e832a1c314be69ec5921b3abc3cd5: Bootstrap starting.
I20260621 05:21:25.932525 10016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ee8c7971e041008b60f1eb07512db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:25.932632 10256 tablet_bootstrap.cc:492] T e6ee8c7971e041008b60f1eb07512db6 P cfa08fd3741046fd93232e135785615c: No bootstrap required, opened a new log
I20260621 05:21:25.932695 10256 ts_tablet_manager.cc:1397] T e6ee8c7971e041008b60f1eb07512db6 P cfa08fd3741046fd93232e135785615c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:21:25.932849 10256 raft_consensus.cc:348] T e6ee8c7971e041008b60f1eb07512db6 P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:25.932924 10256 raft_consensus.cc:374] T e6ee8c7971e041008b60f1eb07512db6 P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:25.932942 10256 raft_consensus.cc:729] T e6ee8c7971e041008b60f1eb07512db6 P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfa08fd3741046fd93232e135785615c, State: Initialized, Role: FOLLOWER
I20260621 05:21:25.932996 10256 consensus_queue.cc:260] T e6ee8c7971e041008b60f1eb07512db6 P cfa08fd3741046fd93232e135785615c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:25.933087 10256 ts_tablet_manager.cc:1428] T e6ee8c7971e041008b60f1eb07512db6 P cfa08fd3741046fd93232e135785615c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:25.933149 10250 tablet_bootstrap.cc:654] T e6ee8c7971e041008b60f1eb07512db6 P 889e832a1c314be69ec5921b3abc3cd5: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:25.933173 10258 tablet_replica.cc:439] T e6ee8c7971e041008b60f1eb07512db6 P cfa08fd3741046fd93232e135785615c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } } }
I20260621 05:21:25.933305 10258 tablet_replica.cc:442] T e6ee8c7971e041008b60f1eb07512db6 P cfa08fd3741046fd93232e135785615c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:21:25.933929 10250 tablet_bootstrap.cc:492] T e6ee8c7971e041008b60f1eb07512db6 P 889e832a1c314be69ec5921b3abc3cd5: No bootstrap required, opened a new log
I20260621 05:21:25.933987 10250 ts_tablet_manager.cc:1397] T e6ee8c7971e041008b60f1eb07512db6 P 889e832a1c314be69ec5921b3abc3cd5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:21:25.934032 10251 tablet_bootstrap.cc:492] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78: Bootstrap starting.
I20260621 05:21:25.934175 10250 raft_consensus.cc:348] T e6ee8c7971e041008b60f1eb07512db6 P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:25.934255 10250 raft_consensus.cc:374] T e6ee8c7971e041008b60f1eb07512db6 P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:25.934284 10250 raft_consensus.cc:729] T e6ee8c7971e041008b60f1eb07512db6 P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889e832a1c314be69ec5921b3abc3cd5, State: Initialized, Role: FOLLOWER
I20260621 05:21:25.934330 10250 consensus_queue.cc:260] T e6ee8c7971e041008b60f1eb07512db6 P 889e832a1c314be69ec5921b3abc3cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:25.934434 10250 ts_tablet_manager.cc:1428] T e6ee8c7971e041008b60f1eb07512db6 P 889e832a1c314be69ec5921b3abc3cd5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:21:25.934446 10251 tablet_bootstrap.cc:654] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:25.934573 10254 tablet_replica.cc:439] T e6ee8c7971e041008b60f1eb07512db6 P 889e832a1c314be69ec5921b3abc3cd5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } } }
I20260621 05:21:25.934667 10254 tablet_replica.cc:442] T e6ee8c7971e041008b60f1eb07512db6 P 889e832a1c314be69ec5921b3abc3cd5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:21:25.935230 10251 tablet_bootstrap.cc:492] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78: No bootstrap required, opened a new log
I20260621 05:21:25.935312 10251 ts_tablet_manager.cc:1397] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260621 05:21:25.935436 10251 raft_consensus.cc:348] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:25.935534 10251 raft_consensus.cc:374] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:25.935562 10251 raft_consensus.cc:729] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a535c9286b64080b815c5e965783f78, State: Initialized, Role: FOLLOWER
I20260621 05:21:25.935611 10251 consensus_queue.cc:260] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:25.935717 10251 ts_tablet_manager.cc:1428] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:25.935781 10255 tablet_replica.cc:439] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } } }
I20260621 05:21:25.935902 10255 tablet_replica.cc:442] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:21:25.968726 10255 raft_consensus.cc:482] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:21:25.968883 10255 raft_consensus.cc:504] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:25.969065 10255 leader_election.cc:288] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfa08fd3741046fd93232e135785615c (127.2.32.1:36237), 889e832a1c314be69ec5921b3abc3cd5 (127.2.32.3:37013)
I20260621 05:21:25.969551 10175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ee8c7971e041008b60f1eb07512db6" candidate_uuid: "0a535c9286b64080b815c5e965783f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889e832a1c314be69ec5921b3abc3cd5" is_pre_election: true
I20260621 05:21:25.969425 9896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ee8c7971e041008b60f1eb07512db6" candidate_uuid: "0a535c9286b64080b815c5e965783f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfa08fd3741046fd93232e135785615c" is_pre_election: true
I20260621 05:21:25.969683 10175 raft_consensus.cc:2454] T e6ee8c7971e041008b60f1eb07512db6 P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a535c9286b64080b815c5e965783f78 in term 0.
I20260621 05:21:25.969739 9896 raft_consensus.cc:2454] T e6ee8c7971e041008b60f1eb07512db6 P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a535c9286b64080b815c5e965783f78 in term 0.
I20260621 05:21:25.969982 9970 leader_election.cc:302] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a535c9286b64080b815c5e965783f78, cfa08fd3741046fd93232e135785615c; no voters:
I20260621 05:21:25.970266 10255 raft_consensus.cc:2781] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:21:25.970325 10255 raft_consensus.cc:482] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:21:25.970363 10255 raft_consensus.cc:3037] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:25.971400 10255 raft_consensus.cc:504] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:25.971581 10255 leader_election.cc:288] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78 [CANDIDATE]: Term 1 election: Requested vote from peers cfa08fd3741046fd93232e135785615c (127.2.32.1:36237), 889e832a1c314be69ec5921b3abc3cd5 (127.2.32.3:37013)
I20260621 05:21:25.972034 10175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ee8c7971e041008b60f1eb07512db6" candidate_uuid: "0a535c9286b64080b815c5e965783f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889e832a1c314be69ec5921b3abc3cd5"
I20260621 05:21:25.972067 9896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ee8c7971e041008b60f1eb07512db6" candidate_uuid: "0a535c9286b64080b815c5e965783f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfa08fd3741046fd93232e135785615c"
I20260621 05:21:25.972154 10175 raft_consensus.cc:3037] T e6ee8c7971e041008b60f1eb07512db6 P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:25.972172 9896 raft_consensus.cc:3037] T e6ee8c7971e041008b60f1eb07512db6 P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:25.972925 9896 raft_consensus.cc:2454] T e6ee8c7971e041008b60f1eb07512db6 P cfa08fd3741046fd93232e135785615c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a535c9286b64080b815c5e965783f78 in term 1.
I20260621 05:21:25.973073 10175 raft_consensus.cc:2454] T e6ee8c7971e041008b60f1eb07512db6 P 889e832a1c314be69ec5921b3abc3cd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a535c9286b64080b815c5e965783f78 in term 1.
I20260621 05:21:25.973146 9970 leader_election.cc:302] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a535c9286b64080b815c5e965783f78, cfa08fd3741046fd93232e135785615c; no voters:
I20260621 05:21:25.973363 10255 raft_consensus.cc:2781] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:21:25.973460 10255 raft_consensus.cc:686] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78 [term 1 LEADER]: Becoming Leader. State: Replica: 0a535c9286b64080b815c5e965783f78, State: Running, Role: LEADER
I20260621 05:21:25.973527 10255 consensus_queue.cc:237] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:25.973925 10260 tablet_replica.cc:439] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78: TxnStatusTablet state changed. Reason: New leader 0a535c9286b64080b815c5e965783f78. Latest consensus state: current_term: 1 leader_uuid: "0a535c9286b64080b815c5e965783f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } } }
I20260621 05:21:25.974017 10260 tablet_replica.cc:442] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:21:25.974202 10292 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 05:21:25.974376 10292 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 05:21:25.974349 9739 catalog_manager.cc:5495] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a535c9286b64080b815c5e965783f78 (127.2.32.2). New cstate: current_term: 1 leader_uuid: "0a535c9286b64080b815c5e965783f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } health_report { overall_health: UNKNOWN } } }
I20260621 05:21:25.981791 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.979440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48770 (local address 127.2.32.2:44405)
0621 05:21:25.979485 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:25.979487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:25.979496 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:21:25.979576 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:25.979577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:25.979589 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:25.979635 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:21:25.979789 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.980252 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.981086 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.981294 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.981380 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.981548 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:21:25.981581 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:21:25.981645 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:25.981708 (+ 63us) server_negotiation.cc:299] Negotiation successful
0621 05:21:25.981727 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:21:25.981845 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.979371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:44405 (local address 127.0.0.1:48770)
0621 05:21:25.979440 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:25.979452 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:25.979504 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:25.979648 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:25.979649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:25.979650 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:21:25.979698 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.979701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.980349 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.980351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.980984 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.980985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.981382 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.981383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.981436 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.981457 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:21:25.981657 (+ 200us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:21:25.981665 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 05:21:25.981707 (+ 42us) client_negotiation.cc:240] Negotiation successful
0621 05:21:25.981764 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260621 05:21:25.982760 10175 raft_consensus.cc:1261] T e6ee8c7971e041008b60f1eb07512db6 P 889e832a1c314be69ec5921b3abc3cd5 [term 1 FOLLOWER]: Refusing update from remote peer 0a535c9286b64080b815c5e965783f78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:21:25.982789 9896 raft_consensus.cc:1261] T e6ee8c7971e041008b60f1eb07512db6 P cfa08fd3741046fd93232e135785615c [term 1 FOLLOWER]: Refusing update from remote peer 0a535c9286b64080b815c5e965783f78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:21:25.983018 10260 consensus_queue.cc:1035] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:21:25.983186 10255 consensus_queue.cc:1035] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:21:25.983716 10254 tablet_replica.cc:439] T e6ee8c7971e041008b60f1eb07512db6 P 889e832a1c314be69ec5921b3abc3cd5: TxnStatusTablet state changed. Reason: New leader 0a535c9286b64080b815c5e965783f78. Latest consensus state: current_term: 1 leader_uuid: "0a535c9286b64080b815c5e965783f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } } }
I20260621 05:21:25.983811 10254 tablet_replica.cc:442] T e6ee8c7971e041008b60f1eb07512db6 P 889e832a1c314be69ec5921b3abc3cd5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:21:25.983803 10258 tablet_replica.cc:439] T e6ee8c7971e041008b60f1eb07512db6 P cfa08fd3741046fd93232e135785615c: TxnStatusTablet state changed. Reason: New leader 0a535c9286b64080b815c5e965783f78. Latest consensus state: current_term: 1 leader_uuid: "0a535c9286b64080b815c5e965783f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } } }
I20260621 05:21:25.983882 10258 tablet_replica.cc:442] T e6ee8c7971e041008b60f1eb07512db6 P cfa08fd3741046fd93232e135785615c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:21:25.984566 10255 tablet_replica.cc:439] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a535c9286b64080b815c5e965783f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } } }
I20260621 05:21:25.984649 10255 tablet_replica.cc:442] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:21:25.984601 10260 tablet_replica.cc:439] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a535c9286b64080b815c5e965783f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } } }
I20260621 05:21:25.984684 10260 tablet_replica.cc:442] T e6ee8c7971e041008b60f1eb07512db6 P 0a535c9286b64080b815c5e965783f78: This TxnStatusTablet replica's current role is: LEADER
I20260621 05:21:25.984704 10263 mvcc.cc:205] Tried to move back new op lower bound from 7299150995383730176 to 7299150995348086784. Current Snapshot: MvccSnapshot[applied={T|T < 7299150995383730176}]
I20260621 05:21:25.985378 10254 tablet_replica.cc:439] T e6ee8c7971e041008b60f1eb07512db6 P 889e832a1c314be69ec5921b3abc3cd5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a535c9286b64080b815c5e965783f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } } }
I20260621 05:21:25.985491 10254 tablet_replica.cc:442] T e6ee8c7971e041008b60f1eb07512db6 P 889e832a1c314be69ec5921b3abc3cd5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:21:25.985945 10258 tablet_replica.cc:439] T e6ee8c7971e041008b60f1eb07512db6 P cfa08fd3741046fd93232e135785615c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a535c9286b64080b815c5e965783f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } } }
I20260621 05:21:25.986029 10258 tablet_replica.cc:442] T e6ee8c7971e041008b60f1eb07512db6 P cfa08fd3741046fd93232e135785615c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 05:21:25.990545 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.986832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33040 (local address 127.2.32.3:37013)
0621 05:21:25.986910 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:25.986913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:25.986925 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:21:25.986981 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:25.986982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:25.987011 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:25.987177 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:21:25.987410 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.988290 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.989250 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.989626 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.989711 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.989904 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:21:25.990043 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:21:25.990144 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:25.990364 (+ 220us) server_negotiation.cc:299] Negotiation successful
0621 05:21:25.990442 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 05:21:25.996901 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.993754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:54303 (local address 127.2.32.2:44405)
0621 05:21:25.993803 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:25.993805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:25.993906 (+ 101us) server_negotiation.cc:407] Connection header received
0621 05:21:25.993979 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:25.993980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:25.993993 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:25.994047 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:21:25.994225 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.994682 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.995835 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.996072 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.996139 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.996503 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:21:25.996528 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:21:25.996571 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:25.996815 (+ 244us) server_negotiation.cc:299] Negotiation successful
0621 05:21:25.996842 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:21:25.997159 10300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:25.993643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:44405 (local address 127.2.32.3:54303)
0621 05:21:25.993828 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:25.993840 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:25.993917 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:25.994069 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:25.994070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:25.994072 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:21:25.994164 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.994165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.994749 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.994750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.995734 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:25.995736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:25.996236 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:25.996237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:25.996360 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:25.996374 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:21:25.996660 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:21:25.996663 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:21:25.996731 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 05:21:25.996789 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260621 05:21:25.999248 9896 raft_consensus.cc:1261] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c [term 1 FOLLOWER]: Refusing update from remote peer 889e832a1c314be69ec5921b3abc3cd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:21:25.999274 10036 raft_consensus.cc:1261] T f43d5e6fcb174aea83dc442f2b9b5e5f P 0a535c9286b64080b815c5e965783f78 [term 1 FOLLOWER]: Refusing update from remote peer 889e832a1c314be69ec5921b3abc3cd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:21:25.999625 10279 consensus_queue.cc:1035] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:21:26.000061 10279 consensus_queue.cc:1035] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:21:26.001471 10264 mvcc.cc:205] Tried to move back new op lower bound from 7299150995451256832 to 7299150994702782464. Current Snapshot: MvccSnapshot[applied={T|T < 7299150995451256832}]
I20260621 05:21:26.007884 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:26.004967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:39833 (local address 127.2.32.3:37013)
0621 05:21:26.005083 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:26.005085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:26.005150 (+ 65us) server_negotiation.cc:407] Connection header received
0621 05:21:26.005230 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:26.005230 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:26.005240 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:26.005279 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:21:26.005472 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.005996 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.007204 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.007398 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.007451 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:26.007682 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:21:26.007705 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:21:26.007757 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:26.007822 (+ 65us) server_negotiation.cc:299] Negotiation successful
0621 05:21:26.007838 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260621 05:21:26.008116 10304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:26.004890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:37013 (local address 127.2.32.2:39833)
0621 05:21:26.005091 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:26.005103 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:26.005174 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:26.005292 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:26.005292 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:26.005294 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:21:26.005358 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:26.005360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.006079 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.006080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:26.007086 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:26.007089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.007470 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.007471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:26.007542 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:26.007556 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:21:26.007766 (+ 210us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:21:26.007767 (+ 1us) client_negotiation.cc:769] Sending connection context
0621 05:21:26.007821 (+ 54us) client_negotiation.cc:240] Negotiation successful
0621 05:21:26.007863 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":117,"threads_started":1}
I20260621 05:21:26.017769 9738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:21:26.017882 9738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:21:26.018496 9738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9: Sending DeleteTablet for 3 replicas of tablet f43d5e6fcb174aea83dc442f2b9b5e5f
I20260621 05:21:26.018724 10016 tablet_service.cc:1514] Processing DeleteTablet for tablet f43d5e6fcb174aea83dc442f2b9b5e5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:21:26 UTC) from {username='oryx', principal='oryx/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:48748
I20260621 05:21:26.018777 9876 tablet_service.cc:1514] Processing DeleteTablet for tablet f43d5e6fcb174aea83dc442f2b9b5e5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:21:26 UTC) from {username='oryx', principal='oryx/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:41638
I20260621 05:21:26.018858 10274 tablet_replica.cc:330] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c: stopping tablet replica
I20260621 05:21:26.018848 10272 tablet_replica.cc:330] T f43d5e6fcb174aea83dc442f2b9b5e5f P 0a535c9286b64080b815c5e965783f78: stopping tablet replica
I20260621 05:21:26.018910 10274 raft_consensus.cc:2229] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:21:26.018918 10272 raft_consensus.cc:2229] T f43d5e6fcb174aea83dc442f2b9b5e5f P 0a535c9286b64080b815c5e965783f78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:21:26.018944 10274 raft_consensus.cc:2258] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:21:26.018953 10272 raft_consensus.cc:2258] T f43d5e6fcb174aea83dc442f2b9b5e5f P 0a535c9286b64080b815c5e965783f78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:21:26.019272 10274 ts_tablet_manager.cc:1905] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:21:26.019311 10156 tablet_service.cc:1514] Processing DeleteTablet for tablet f43d5e6fcb174aea83dc442f2b9b5e5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:21:26 UTC) from {username='oryx', principal='oryx/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:33026
I20260621 05:21:26.019383 10273 tablet_replica.cc:330] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5: stopping tablet replica
I20260621 05:21:26.019423 10273 raft_consensus.cc:2229] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:21:26.019510 10273 raft_consensus.cc:2258] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:21:26.020700 10272 ts_tablet_manager.cc:1905] T f43d5e6fcb174aea83dc442f2b9b5e5f P 0a535c9286b64080b815c5e965783f78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:21:26.021921 10273 ts_tablet_manager.cc:1905] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:21:26.022327 10272 ts_tablet_manager.cc:1918] T f43d5e6fcb174aea83dc442f2b9b5e5f P 0a535c9286b64080b815c5e965783f78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:21:26.022404 10272 log.cc:1196] T f43d5e6fcb174aea83dc442f2b9b5e5f P 0a535c9286b64080b815c5e965783f78: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/f43d5e6fcb174aea83dc442f2b9b5e5f
I20260621 05:21:26.022614 10272 ts_tablet_manager.cc:1939] T f43d5e6fcb174aea83dc442f2b9b5e5f P 0a535c9286b64080b815c5e965783f78: Deleting consensus metadata
I20260621 05:21:26.022969 9724 catalog_manager.cc:4841] TS 0a535c9286b64080b815c5e965783f78 (127.2.32.2:44405): tablet f43d5e6fcb174aea83dc442f2b9b5e5f (table test-table [id=5da8ff0591b24f84a5572034d29dd2ab]) successfully deleted
I20260621 05:21:26.023605 10273 ts_tablet_manager.cc:1918] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:21:26.023667 10273 log.cc:1196] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/f43d5e6fcb174aea83dc442f2b9b5e5f
I20260621 05:21:26.023885 10273 ts_tablet_manager.cc:1939] T f43d5e6fcb174aea83dc442f2b9b5e5f P 889e832a1c314be69ec5921b3abc3cd5: Deleting consensus metadata
I20260621 05:21:26.024214 9726 catalog_manager.cc:4841] TS 889e832a1c314be69ec5921b3abc3cd5 (127.2.32.3:37013): tablet f43d5e6fcb174aea83dc442f2b9b5e5f (table test-table [id=5da8ff0591b24f84a5572034d29dd2ab]) successfully deleted
I20260621 05:21:26.024830 10274 ts_tablet_manager.cc:1918] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:21:26.024896 10274 log.cc:1196] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-0/wal/wals/f43d5e6fcb174aea83dc442f2b9b5e5f
I20260621 05:21:26.025126 10274 ts_tablet_manager.cc:1939] T f43d5e6fcb174aea83dc442f2b9b5e5f P cfa08fd3741046fd93232e135785615c: Deleting consensus metadata
I20260621 05:21:26.025426 9726 catalog_manager.cc:4841] TS cfa08fd3741046fd93232e135785615c (127.2.32.1:36237): tablet f43d5e6fcb174aea83dc442f2b9b5e5f (table test-table [id=5da8ff0591b24f84a5572034d29dd2ab]) successfully deleted
Jun 21 05:21:26 dist-test-slave-b357 krb5kdc[9694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 05:21:26 dist-test-slave-b357 krb5kdc[9694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:26 dist-test-slave-b357 krb5kdc[9694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 05:21:26.052960 10287 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:26.045731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35618 (local address 127.2.32.62:43427)
0621 05:21:26.045800 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:26.045803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:26.045925 (+ 122us) server_negotiation.cc:407] Connection header received
0621 05:21:26.046044 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:26.046045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:26.046064 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:26.046173 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:26.046994 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.047848 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.048994 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.049314 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.049356 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:26.052880 (+ 3524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35618: BlockingRecv error: recv got EOF from 127.0.0.1:35618 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 21 05:21:26 dist-test-slave-b357 krb5kdc[9694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM
I20260621 05:21:26.062824 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:26.053659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.2.32.62:43427)
0621 05:21:26.053700 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:26.053702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:26.054062 (+ 360us) server_negotiation.cc:407] Connection header received
0621 05:21:26.054096 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:26.054097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:26.054113 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:26.054185 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:26.054846 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.055858 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.056933 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.057239 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.057317 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:26.059914 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:26.059916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:26.059916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:21:26.059924 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:26.061544 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:26.061929 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:26.061930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:26.061931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:26.061991 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:26.062257 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:26.062259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:26.062259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:26.062399 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:26.062502 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:26.062624 (+ 122us) server_negotiation.cc:299] Negotiation successful
0621 05:21:26.062712 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:21:26.064471 9738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:21:26.064642 9738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:21:26.065920 9876 tablet_service.cc:1467] Processing CreateTablet for tablet b91ff9cf411c46d79fe6e51cd98c8581 (DEFAULT_TABLE table=test-table [id=d554afe2a401409ea8a635279772e6dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:21:26.065941 10156 tablet_service.cc:1467] Processing CreateTablet for tablet b91ff9cf411c46d79fe6e51cd98c8581 (DEFAULT_TABLE table=test-table [id=d554afe2a401409ea8a635279772e6dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:21:26.066057 9876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91ff9cf411c46d79fe6e51cd98c8581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:26.066072 10156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91ff9cf411c46d79fe6e51cd98c8581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:26.067443 10256 tablet_bootstrap.cc:492] T b91ff9cf411c46d79fe6e51cd98c8581 P cfa08fd3741046fd93232e135785615c: Bootstrap starting.
I20260621 05:21:26.067487 10250 tablet_bootstrap.cc:492] T b91ff9cf411c46d79fe6e51cd98c8581 P 889e832a1c314be69ec5921b3abc3cd5: Bootstrap starting.
I20260621 05:21:26.067808 10256 tablet_bootstrap.cc:654] T b91ff9cf411c46d79fe6e51cd98c8581 P cfa08fd3741046fd93232e135785615c: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:26.067950 10250 tablet_bootstrap.cc:654] T b91ff9cf411c46d79fe6e51cd98c8581 P 889e832a1c314be69ec5921b3abc3cd5: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:26.068646 10016 tablet_service.cc:1467] Processing CreateTablet for tablet b91ff9cf411c46d79fe6e51cd98c8581 (DEFAULT_TABLE table=test-table [id=d554afe2a401409ea8a635279772e6dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:21:26.068760 10016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91ff9cf411c46d79fe6e51cd98c8581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:26.068755 10250 tablet_bootstrap.cc:492] T b91ff9cf411c46d79fe6e51cd98c8581 P 889e832a1c314be69ec5921b3abc3cd5: No bootstrap required, opened a new log
I20260621 05:21:26.068799 10250 ts_tablet_manager.cc:1397] T b91ff9cf411c46d79fe6e51cd98c8581 P 889e832a1c314be69ec5921b3abc3cd5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:26.068981 10250 raft_consensus.cc:348] T b91ff9cf411c46d79fe6e51cd98c8581 P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:26.069046 10250 raft_consensus.cc:374] T b91ff9cf411c46d79fe6e51cd98c8581 P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:26.069063 10250 raft_consensus.cc:729] T b91ff9cf411c46d79fe6e51cd98c8581 P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889e832a1c314be69ec5921b3abc3cd5, State: Initialized, Role: FOLLOWER
I20260621 05:21:26.069103 10250 consensus_queue.cc:260] T b91ff9cf411c46d79fe6e51cd98c8581 P 889e832a1c314be69ec5921b3abc3cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:26.069181 10250 ts_tablet_manager.cc:1428] T b91ff9cf411c46d79fe6e51cd98c8581 P 889e832a1c314be69ec5921b3abc3cd5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:21:26.069976 10251 tablet_bootstrap.cc:492] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78: Bootstrap starting.
I20260621 05:21:26.070250 10251 tablet_bootstrap.cc:654] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:26.071023 10251 tablet_bootstrap.cc:492] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78: No bootstrap required, opened a new log
I20260621 05:21:26.071133 10251 ts_tablet_manager.cc:1397] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260621 05:21:26.071342 10251 raft_consensus.cc:348] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:26.071439 10251 raft_consensus.cc:374] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:26.071473 10251 raft_consensus.cc:729] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a535c9286b64080b815c5e965783f78, State: Initialized, Role: FOLLOWER
I20260621 05:21:26.071626 10251 consensus_queue.cc:260] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:26.071784 10251 ts_tablet_manager.cc:1428] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260621 05:21:26.072739 10256 tablet_bootstrap.cc:492] T b91ff9cf411c46d79fe6e51cd98c8581 P cfa08fd3741046fd93232e135785615c: No bootstrap required, opened a new log
I20260621 05:21:26.072808 10256 ts_tablet_manager.cc:1397] T b91ff9cf411c46d79fe6e51cd98c8581 P cfa08fd3741046fd93232e135785615c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260621 05:21:26.072934 10256 raft_consensus.cc:348] T b91ff9cf411c46d79fe6e51cd98c8581 P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:26.073004 10256 raft_consensus.cc:374] T b91ff9cf411c46d79fe6e51cd98c8581 P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:26.073024 10256 raft_consensus.cc:729] T b91ff9cf411c46d79fe6e51cd98c8581 P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfa08fd3741046fd93232e135785615c, State: Initialized, Role: FOLLOWER
I20260621 05:21:26.073062 10256 consensus_queue.cc:260] T b91ff9cf411c46d79fe6e51cd98c8581 P cfa08fd3741046fd93232e135785615c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:26.073133 10256 ts_tablet_manager.cc:1428] T b91ff9cf411c46d79fe6e51cd98c8581 P cfa08fd3741046fd93232e135785615c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:21:26.166481 10297 raft_consensus.cc:482] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:21:26.166625 10297 raft_consensus.cc:504] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:26.166867 10297 leader_election.cc:288] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfa08fd3741046fd93232e135785615c (127.2.32.1:36237), 889e832a1c314be69ec5921b3abc3cd5 (127.2.32.3:37013)
I20260621 05:21:26.167196 9896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91ff9cf411c46d79fe6e51cd98c8581" candidate_uuid: "0a535c9286b64080b815c5e965783f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfa08fd3741046fd93232e135785615c" is_pre_election: true
I20260621 05:21:26.167356 9896 raft_consensus.cc:2454] T b91ff9cf411c46d79fe6e51cd98c8581 P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a535c9286b64080b815c5e965783f78 in term 0.
I20260621 05:21:26.167447 10175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91ff9cf411c46d79fe6e51cd98c8581" candidate_uuid: "0a535c9286b64080b815c5e965783f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889e832a1c314be69ec5921b3abc3cd5" is_pre_election: true
I20260621 05:21:26.167541 9970 leader_election.cc:302] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a535c9286b64080b815c5e965783f78, cfa08fd3741046fd93232e135785615c; no voters:
I20260621 05:21:26.167572 10175 raft_consensus.cc:2454] T b91ff9cf411c46d79fe6e51cd98c8581 P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a535c9286b64080b815c5e965783f78 in term 0.
I20260621 05:21:26.167783 10297 raft_consensus.cc:2781] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:21:26.167840 10297 raft_consensus.cc:482] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:21:26.167860 10297 raft_consensus.cc:3037] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:26.168916 10297 raft_consensus.cc:504] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:26.169064 10297 leader_election.cc:288] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [CANDIDATE]: Term 1 election: Requested vote from peers cfa08fd3741046fd93232e135785615c (127.2.32.1:36237), 889e832a1c314be69ec5921b3abc3cd5 (127.2.32.3:37013)
I20260621 05:21:26.169252 9896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91ff9cf411c46d79fe6e51cd98c8581" candidate_uuid: "0a535c9286b64080b815c5e965783f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfa08fd3741046fd93232e135785615c"
I20260621 05:21:26.169256 10175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91ff9cf411c46d79fe6e51cd98c8581" candidate_uuid: "0a535c9286b64080b815c5e965783f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889e832a1c314be69ec5921b3abc3cd5"
I20260621 05:21:26.169338 9896 raft_consensus.cc:3037] T b91ff9cf411c46d79fe6e51cd98c8581 P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:26.169349 10175 raft_consensus.cc:3037] T b91ff9cf411c46d79fe6e51cd98c8581 P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:26.170131 10175 raft_consensus.cc:2454] T b91ff9cf411c46d79fe6e51cd98c8581 P 889e832a1c314be69ec5921b3abc3cd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a535c9286b64080b815c5e965783f78 in term 1.
I20260621 05:21:26.170233 9896 raft_consensus.cc:2454] T b91ff9cf411c46d79fe6e51cd98c8581 P cfa08fd3741046fd93232e135785615c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a535c9286b64080b815c5e965783f78 in term 1.
I20260621 05:21:26.170322 9970 leader_election.cc:302] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a535c9286b64080b815c5e965783f78, 889e832a1c314be69ec5921b3abc3cd5; no voters:
I20260621 05:21:26.170513 10297 raft_consensus.cc:2781] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:21:26.170593 10297 raft_consensus.cc:686] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [term 1 LEADER]: Becoming Leader. State: Replica: 0a535c9286b64080b815c5e965783f78, State: Running, Role: LEADER
I20260621 05:21:26.170656 10297 consensus_queue.cc:237] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:26.171599 9738 catalog_manager.cc:5495] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a535c9286b64080b815c5e965783f78 (127.2.32.2). New cstate: current_term: 1 leader_uuid: "0a535c9286b64080b815c5e965783f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } health_report { overall_health: UNKNOWN } } }
I20260621 05:21:26.191051 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:26.188796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48772 (local address 127.2.32.2:44405)
0621 05:21:26.188887 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:26.188890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:26.188898 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:21:26.188923 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:26.188923 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:26.188934 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:26.189004 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:21:26.189126 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.189595 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.190345 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.190545 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.190621 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:26.190781 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:21:26.190818 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:21:26.190879 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:26.190937 (+ 58us) server_negotiation.cc:299] Negotiation successful
0621 05:21:26.190954 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 05:21:26.192080 9896 raft_consensus.cc:1261] T b91ff9cf411c46d79fe6e51cd98c8581 P cfa08fd3741046fd93232e135785615c [term 1 FOLLOWER]: Refusing update from remote peer 0a535c9286b64080b815c5e965783f78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:21:26.192075 10175 raft_consensus.cc:1261] T b91ff9cf411c46d79fe6e51cd98c8581 P 889e832a1c314be69ec5921b3abc3cd5 [term 1 FOLLOWER]: Refusing update from remote peer 0a535c9286b64080b815c5e965783f78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:21:26.192387 10307 consensus_queue.cc:1035] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:21:26.192487 10297 consensus_queue.cc:1035] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:21:26.196390 9739 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:21:26.196481 9739 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:21:26.197288 9739 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9: Sending DeleteTablet for 3 replicas of tablet b91ff9cf411c46d79fe6e51cd98c8581
I20260621 05:21:26.197557 10016 tablet_service.cc:1514] Processing DeleteTablet for tablet b91ff9cf411c46d79fe6e51cd98c8581 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:21:26 UTC) from {username='oryx', principal='oryx/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:48748
I20260621 05:21:26.197683 10272 tablet_replica.cc:330] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78: stopping tablet replica
I20260621 05:21:26.197695 10156 tablet_service.cc:1514] Processing DeleteTablet for tablet b91ff9cf411c46d79fe6e51cd98c8581 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:21:26 UTC) from {username='oryx', principal='oryx/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:33026
I20260621 05:21:26.197757 10272 raft_consensus.cc:2229] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:21:26.197809 10273 tablet_replica.cc:330] T b91ff9cf411c46d79fe6e51cd98c8581 P 889e832a1c314be69ec5921b3abc3cd5: stopping tablet replica
I20260621 05:21:26.197835 10272 raft_consensus.cc:2258] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:21:26.197865 10273 raft_consensus.cc:2229] T b91ff9cf411c46d79fe6e51cd98c8581 P 889e832a1c314be69ec5921b3abc3cd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:21:26.197904 10273 raft_consensus.cc:2258] T b91ff9cf411c46d79fe6e51cd98c8581 P 889e832a1c314be69ec5921b3abc3cd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:21:26.197891 9876 tablet_service.cc:1514] Processing DeleteTablet for tablet b91ff9cf411c46d79fe6e51cd98c8581 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:21:26 UTC) from {username='oryx', principal='oryx/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:41638
I20260621 05:21:26.197981 10274 tablet_replica.cc:330] T b91ff9cf411c46d79fe6e51cd98c8581 P cfa08fd3741046fd93232e135785615c: stopping tablet replica
I20260621 05:21:26.197945 9739 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260621 05:21:26.198048 10274 raft_consensus.cc:2229] T b91ff9cf411c46d79fe6e51cd98c8581 P cfa08fd3741046fd93232e135785615c [term 1 FOLLOWER]: Raft consensus shutting down.
W20260621 05:21:26.198082 9739 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:21:26.198107 10274 raft_consensus.cc:2258] T b91ff9cf411c46d79fe6e51cd98c8581 P cfa08fd3741046fd93232e135785615c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:21:26.198319 10273 ts_tablet_manager.cc:1905] T b91ff9cf411c46d79fe6e51cd98c8581 P 889e832a1c314be69ec5921b3abc3cd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:21:26.198390 10274 ts_tablet_manager.cc:1905] T b91ff9cf411c46d79fe6e51cd98c8581 P cfa08fd3741046fd93232e135785615c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:21:26.198442 10272 ts_tablet_manager.cc:1905] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:21:26.199797 10272 ts_tablet_manager.cc:1918] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:21:26.199849 10274 ts_tablet_manager.cc:1918] T b91ff9cf411c46d79fe6e51cd98c8581 P cfa08fd3741046fd93232e135785615c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:21:26.199849 10273 ts_tablet_manager.cc:1918] T b91ff9cf411c46d79fe6e51cd98c8581 P 889e832a1c314be69ec5921b3abc3cd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:21:26.199872 10272 log.cc:1196] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/b91ff9cf411c46d79fe6e51cd98c8581
I20260621 05:21:26.199900 10274 log.cc:1196] T b91ff9cf411c46d79fe6e51cd98c8581 P cfa08fd3741046fd93232e135785615c: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-0/wal/wals/b91ff9cf411c46d79fe6e51cd98c8581
I20260621 05:21:26.199900 10273 log.cc:1196] T b91ff9cf411c46d79fe6e51cd98c8581 P 889e832a1c314be69ec5921b3abc3cd5: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/b91ff9cf411c46d79fe6e51cd98c8581
I20260621 05:21:26.200098 10272 ts_tablet_manager.cc:1939] T b91ff9cf411c46d79fe6e51cd98c8581 P 0a535c9286b64080b815c5e965783f78: Deleting consensus metadata
I20260621 05:21:26.200126 10273 ts_tablet_manager.cc:1939] T b91ff9cf411c46d79fe6e51cd98c8581 P 889e832a1c314be69ec5921b3abc3cd5: Deleting consensus metadata
I20260621 05:21:26.200139 10274 ts_tablet_manager.cc:1939] T b91ff9cf411c46d79fe6e51cd98c8581 P cfa08fd3741046fd93232e135785615c: Deleting consensus metadata
I20260621 05:21:26.200464 9724 catalog_manager.cc:4841] TS 0a535c9286b64080b815c5e965783f78 (127.2.32.2:44405): tablet b91ff9cf411c46d79fe6e51cd98c8581 (table test-table [id=d554afe2a401409ea8a635279772e6dc]) successfully deleted
I20260621 05:21:26.200536 9726 catalog_manager.cc:4841] TS 889e832a1c314be69ec5921b3abc3cd5 (127.2.32.3:37013): tablet b91ff9cf411c46d79fe6e51cd98c8581 (table test-table [id=d554afe2a401409ea8a635279772e6dc]) successfully deleted
I20260621 05:21:26.200610 9726 catalog_manager.cc:4841] TS cfa08fd3741046fd93232e135785615c (127.2.32.1:36237): tablet b91ff9cf411c46d79fe6e51cd98c8581 (table test-table [id=d554afe2a401409ea8a635279772e6dc]) successfully deleted
I20260621 05:21:26.201318 9876 tablet_service.cc:1467] Processing CreateTablet for tablet 49c1c2f4282b4679bad023eaf121456b (DEFAULT_TABLE table=test-table [id=dc2c0f7400774e7b8f40bb25475912df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:21:26.201337 10156 tablet_service.cc:1467] Processing CreateTablet for tablet 49c1c2f4282b4679bad023eaf121456b (DEFAULT_TABLE table=test-table [id=dc2c0f7400774e7b8f40bb25475912df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:21:26.201449 9876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49c1c2f4282b4679bad023eaf121456b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:26.201452 10156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49c1c2f4282b4679bad023eaf121456b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:26.202230 10256 tablet_bootstrap.cc:492] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c: Bootstrap starting.
I20260621 05:21:26.202251 10250 tablet_bootstrap.cc:492] T 49c1c2f4282b4679bad023eaf121456b P 889e832a1c314be69ec5921b3abc3cd5: Bootstrap starting.
I20260621 05:21:26.202301 10016 tablet_service.cc:1467] Processing CreateTablet for tablet 49c1c2f4282b4679bad023eaf121456b (DEFAULT_TABLE table=test-table [id=dc2c0f7400774e7b8f40bb25475912df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:21:26.202401 10016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49c1c2f4282b4679bad023eaf121456b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:26.202553 10250 tablet_bootstrap.cc:654] T 49c1c2f4282b4679bad023eaf121456b P 889e832a1c314be69ec5921b3abc3cd5: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:26.202651 10256 tablet_bootstrap.cc:654] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:26.203346 10256 tablet_bootstrap.cc:492] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c: No bootstrap required, opened a new log
I20260621 05:21:26.203397 10256 ts_tablet_manager.cc:1397] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:26.203529 10256 raft_consensus.cc:348] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:26.203572 10256 raft_consensus.cc:374] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:26.203585 10256 raft_consensus.cc:729] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfa08fd3741046fd93232e135785615c, State: Initialized, Role: FOLLOWER
I20260621 05:21:26.203583 10250 tablet_bootstrap.cc:492] T 49c1c2f4282b4679bad023eaf121456b P 889e832a1c314be69ec5921b3abc3cd5: No bootstrap required, opened a new log
I20260621 05:21:26.203613 10256 consensus_queue.cc:260] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:26.203636 10250 ts_tablet_manager.cc:1397] T 49c1c2f4282b4679bad023eaf121456b P 889e832a1c314be69ec5921b3abc3cd5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:26.203677 10256 ts_tablet_manager.cc:1428] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:26.203752 10250 raft_consensus.cc:348] T 49c1c2f4282b4679bad023eaf121456b P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:26.203835 10250 raft_consensus.cc:374] T 49c1c2f4282b4679bad023eaf121456b P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:26.203864 10250 raft_consensus.cc:729] T 49c1c2f4282b4679bad023eaf121456b P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889e832a1c314be69ec5921b3abc3cd5, State: Initialized, Role: FOLLOWER
I20260621 05:21:26.203869 10251 tablet_bootstrap.cc:492] T 49c1c2f4282b4679bad023eaf121456b P 0a535c9286b64080b815c5e965783f78: Bootstrap starting.
I20260621 05:21:26.203907 10250 consensus_queue.cc:260] T 49c1c2f4282b4679bad023eaf121456b P 889e832a1c314be69ec5921b3abc3cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:26.203995 10250 ts_tablet_manager.cc:1428] T 49c1c2f4282b4679bad023eaf121456b P 889e832a1c314be69ec5921b3abc3cd5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:26.204162 10251 tablet_bootstrap.cc:654] T 49c1c2f4282b4679bad023eaf121456b P 0a535c9286b64080b815c5e965783f78: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:26.204720 10251 tablet_bootstrap.cc:492] T 49c1c2f4282b4679bad023eaf121456b P 0a535c9286b64080b815c5e965783f78: No bootstrap required, opened a new log
I20260621 05:21:26.204782 10251 ts_tablet_manager.cc:1397] T 49c1c2f4282b4679bad023eaf121456b P 0a535c9286b64080b815c5e965783f78: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260621 05:21:26.204881 10251 raft_consensus.cc:348] T 49c1c2f4282b4679bad023eaf121456b P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:26.204942 10251 raft_consensus.cc:374] T 49c1c2f4282b4679bad023eaf121456b P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:26.204964 10251 raft_consensus.cc:729] T 49c1c2f4282b4679bad023eaf121456b P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a535c9286b64080b815c5e965783f78, State: Initialized, Role: FOLLOWER
I20260621 05:21:26.205016 10251 consensus_queue.cc:260] T 49c1c2f4282b4679bad023eaf121456b P 0a535c9286b64080b815c5e965783f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:26.205117 10251 ts_tablet_manager.cc:1428] T 49c1c2f4282b4679bad023eaf121456b P 0a535c9286b64080b815c5e965783f78: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:26.459255 10258 raft_consensus.cc:482] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:21:26.459372 10258 raft_consensus.cc:504] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:26.459703 10258 leader_election.cc:288] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a535c9286b64080b815c5e965783f78 (127.2.32.2:44405), 889e832a1c314be69ec5921b3abc3cd5 (127.2.32.3:37013)
I20260621 05:21:26.462738 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:26.460013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:37933 (local address 127.2.32.2:44405)
0621 05:21:26.460079 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:26.460082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:26.460156 (+ 74us) server_negotiation.cc:407] Connection header received
0621 05:21:26.460238 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:26.460239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:26.460251 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:26.460328 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:21:26.460494 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.460979 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.461958 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.462382 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.462459 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:26.462472 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:26.462626 (+ 154us) server_negotiation.cc:299] Negotiation successful
0621 05:21:26.462658 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:21:26.462749 10331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:26.459910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:44405 (local address 127.2.32.1:37933)
0621 05:21:26.460061 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:26.460076 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:26.460151 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:26.460339 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:26.460340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:26.460343 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:21:26.460418 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:26.460420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.461085 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.461086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:26.461868 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:26.461870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.462459 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.462460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:26.462531 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:26.462535 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:21:26.462617 (+ 82us) client_negotiation.cc:240] Negotiation successful
0621 05:21:26.462656 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":65,"threads_started":1}
I20260621 05:21:26.462827 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:26.460003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:33179 (local address 127.2.32.3:37013)
0621 05:21:26.460079 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:26.460082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:26.460285 (+ 203us) server_negotiation.cc:407] Connection header received
0621 05:21:26.460373 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:26.460374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:26.460383 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:26.460461 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:21:26.460612 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.461069 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.462013 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.462470 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.462549 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:26.462560 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:26.462729 (+ 169us) server_negotiation.cc:299] Negotiation successful
0621 05:21:26.462761 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 05:21:26.462838 10332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:26.459910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:37013 (local address 127.2.32.1:33179)
0621 05:21:26.460212 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:26.460221 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:26.460299 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:26.460467 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:26.460467 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:26.460469 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:21:26.460545 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:26.460547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.461167 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.461168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:26.461915 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:26.461916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.462568 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.462569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:26.462644 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:26.462647 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:21:26.462721 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 05:21:26.462757 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":184,"thread_start_us":57,"threads_started":1}
I20260621 05:21:26.463178 10036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c1c2f4282b4679bad023eaf121456b" candidate_uuid: "cfa08fd3741046fd93232e135785615c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a535c9286b64080b815c5e965783f78" is_pre_election: true
I20260621 05:21:26.463297 10036 raft_consensus.cc:2454] T 49c1c2f4282b4679bad023eaf121456b P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfa08fd3741046fd93232e135785615c in term 0.
I20260621 05:21:26.463249 10175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c1c2f4282b4679bad023eaf121456b" candidate_uuid: "cfa08fd3741046fd93232e135785615c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889e832a1c314be69ec5921b3abc3cd5" is_pre_election: true
I20260621 05:21:26.463366 10175 raft_consensus.cc:2454] T 49c1c2f4282b4679bad023eaf121456b P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfa08fd3741046fd93232e135785615c in term 0.
I20260621 05:21:26.463508 9828 leader_election.cc:302] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a535c9286b64080b815c5e965783f78, cfa08fd3741046fd93232e135785615c; no voters:
I20260621 05:21:26.463663 10258 raft_consensus.cc:2781] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:21:26.463738 10258 raft_consensus.cc:482] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:21:26.463773 10258 raft_consensus.cc:3037] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:26.464442 10258 raft_consensus.cc:504] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:26.464586 10258 leader_election.cc:288] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [CANDIDATE]: Term 1 election: Requested vote from peers 0a535c9286b64080b815c5e965783f78 (127.2.32.2:44405), 889e832a1c314be69ec5921b3abc3cd5 (127.2.32.3:37013)
I20260621 05:21:26.464778 10175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c1c2f4282b4679bad023eaf121456b" candidate_uuid: "cfa08fd3741046fd93232e135785615c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889e832a1c314be69ec5921b3abc3cd5"
I20260621 05:21:26.464815 10036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c1c2f4282b4679bad023eaf121456b" candidate_uuid: "cfa08fd3741046fd93232e135785615c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a535c9286b64080b815c5e965783f78"
I20260621 05:21:26.464859 10175 raft_consensus.cc:3037] T 49c1c2f4282b4679bad023eaf121456b P 889e832a1c314be69ec5921b3abc3cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:26.464901 10036 raft_consensus.cc:3037] T 49c1c2f4282b4679bad023eaf121456b P 0a535c9286b64080b815c5e965783f78 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:26.465440 10175 raft_consensus.cc:2454] T 49c1c2f4282b4679bad023eaf121456b P 889e832a1c314be69ec5921b3abc3cd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfa08fd3741046fd93232e135785615c in term 1.
I20260621 05:21:26.465646 10036 raft_consensus.cc:2454] T 49c1c2f4282b4679bad023eaf121456b P 0a535c9286b64080b815c5e965783f78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfa08fd3741046fd93232e135785615c in term 1.
I20260621 05:21:26.465710 9830 leader_election.cc:302] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889e832a1c314be69ec5921b3abc3cd5, cfa08fd3741046fd93232e135785615c; no voters:
I20260621 05:21:26.465795 10258 raft_consensus.cc:2781] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:21:26.465909 10258 raft_consensus.cc:686] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [term 1 LEADER]: Becoming Leader. State: Replica: cfa08fd3741046fd93232e135785615c, State: Running, Role: LEADER
I20260621 05:21:26.466022 10258 consensus_queue.cc:237] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } }
I20260621 05:21:26.466740 9738 catalog_manager.cc:5495] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c reported cstate change: term changed from 0 to 1, leader changed from <none> to cfa08fd3741046fd93232e135785615c (127.2.32.1). New cstate: current_term: 1 leader_uuid: "cfa08fd3741046fd93232e135785615c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa08fd3741046fd93232e135785615c" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 } health_report { overall_health: UNKNOWN } } }
I20260621 05:21:26.495769 10335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:26.492369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41654 (local address 127.2.32.1:36237)
0621 05:21:26.492609 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:26.492611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:26.492620 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:21:26.492664 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:26.492664 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:26.492688 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:26.492779 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:21:26.492963 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.493722 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.494639 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.494902 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.494976 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:26.495278 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:21:26.495395 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:21:26.495465 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:26.495506 (+ 41us) server_negotiation.cc:299] Negotiation successful
0621 05:21:26.495632 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":147,"threads_started":1}
I20260621 05:21:26.501222 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:26.498744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:41853 (local address 127.2.32.2:44405)
0621 05:21:26.498797 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:26.498799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:26.498806 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:21:26.498828 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:26.498829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:26.498837 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:26.498887 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:21:26.499091 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.499599 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.500398 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.500627 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.500718 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:26.500942 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:21:26.500973 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:21:26.501027 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:26.501147 (+ 120us) server_negotiation.cc:299] Negotiation successful
0621 05:21:26.501171 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:21:26.501246 10338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:26.498546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:44405 (local address 127.2.32.1:41853)
0621 05:21:26.498714 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:26.498724 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:26.498768 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:26.498956 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:26.498957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:26.498959 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:21:26.499017 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:26.499018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.499697 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.499698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:26.500272 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:26.500274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.500732 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.500733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:26.500792 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:26.500804 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:21:26.501077 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:21:26.501080 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:21:26.501143 (+ 63us) client_negotiation.cc:240] Negotiation successful
0621 05:21:26.501183 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260621 05:21:26.503612 10175 raft_consensus.cc:1261] T 49c1c2f4282b4679bad023eaf121456b P 889e832a1c314be69ec5921b3abc3cd5 [term 1 FOLLOWER]: Refusing update from remote peer cfa08fd3741046fd93232e135785615c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:21:26.503656 10036 raft_consensus.cc:1261] T 49c1c2f4282b4679bad023eaf121456b P 0a535c9286b64080b815c5e965783f78 [term 1 FOLLOWER]: Refusing update from remote peer cfa08fd3741046fd93232e135785615c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:21:26.503876 10333 consensus_queue.cc:1035] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a535c9286b64080b815c5e965783f78" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:21:26.504020 10258 consensus_queue.cc:1035] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [LEADER]: Connected to new peer: Peer: permanent_uuid: "889e832a1c314be69ec5921b3abc3cd5" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:21:26.513697 10335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:26.509692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:39019 (local address 127.2.32.1:36237)
0621 05:21:26.509741 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:26.509743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:26.509753 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:21:26.509782 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:26.509783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:26.509794 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:26.509851 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:21:26.510107 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.510910 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.512414 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.512916 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.513086 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:26.513284 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:21:26.513365 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:21:26.513419 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:26.513565 (+ 146us) server_negotiation.cc:299] Negotiation successful
0621 05:21:26.513610 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260621 05:21:26.513778 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:26.509514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:36237 (local address 127.2.32.2:39019)
0621 05:21:26.509627 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:26.509641 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:26.509704 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:26.509854 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:26.509856 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:26.509857 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 05:21:26.510041 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:26.510044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.510993 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.510995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:26.512304 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:26.512314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:26.513098 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:26.513100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:26.513183 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:26.513190 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 05:21:26.513421 (+ 231us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 05:21:26.513424 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:21:26.513525 (+ 101us) client_negotiation.cc:240] Negotiation successful
0621 05:21:26.513637 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":67,"threads_started":1}
I20260621 05:21:26.535529 9739 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:21:26.535646 9739 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:21:26.536552 9739 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8aa2a63cb3f94a838d32185a18ab7fe9: Sending DeleteTablet for 3 replicas of tablet 49c1c2f4282b4679bad023eaf121456b
I20260621 05:21:26.536913 10016 tablet_service.cc:1514] Processing DeleteTablet for tablet 49c1c2f4282b4679bad023eaf121456b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:21:26 UTC) from {username='oryx', principal='oryx/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:48748
I20260621 05:21:26.536996 9876 tablet_service.cc:1514] Processing DeleteTablet for tablet 49c1c2f4282b4679bad023eaf121456b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:21:26 UTC) from {username='oryx', principal='oryx/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:41638
I20260621 05:21:26.537030 10272 tablet_replica.cc:330] T 49c1c2f4282b4679bad023eaf121456b P 0a535c9286b64080b815c5e965783f78: stopping tablet replica
I20260621 05:21:26.537096 10274 tablet_replica.cc:330] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c: stopping tablet replica
I20260621 05:21:26.537107 10272 raft_consensus.cc:2229] T 49c1c2f4282b4679bad023eaf121456b P 0a535c9286b64080b815c5e965783f78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:21:26.537143 10274 raft_consensus.cc:2229] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:21:26.537149 10272 raft_consensus.cc:2258] T 49c1c2f4282b4679bad023eaf121456b P 0a535c9286b64080b815c5e965783f78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:21:26.537256 10274 raft_consensus.cc:2258] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:21:26.537600 10272 ts_tablet_manager.cc:1905] T 49c1c2f4282b4679bad023eaf121456b P 0a535c9286b64080b815c5e965783f78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:21:26.537658 10274 ts_tablet_manager.cc:1905] T 49c1c2f4282b4679bad023eaf121456b P cfa08fd3741046fd93232e135785615c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:21:26.537802 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 9811
I20260621 05:21:26.539449 10272 ts_tablet_manager.cc:1918] T 49c1c2f4282b4679bad023eaf121456b P 0a535c9286b64080b815c5e965783f78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:21:26.539505 10272 log.cc:1196] T 49c1c2f4282b4679bad023eaf121456b P 0a535c9286b64080b815c5e965783f78: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/49c1c2f4282b4679bad023eaf121456b
I20260621 05:21:26.539716 10272 ts_tablet_manager.cc:1939] T 49c1c2f4282b4679bad023eaf121456b P 0a535c9286b64080b815c5e965783f78: Deleting consensus metadata
I20260621 05:21:26.540040 9724 catalog_manager.cc:4841] TS 0a535c9286b64080b815c5e965783f78 (127.2.32.2:44405): tablet 49c1c2f4282b4679bad023eaf121456b (table test-table [id=dc2c0f7400774e7b8f40bb25475912df]) successfully deleted
I20260621 05:21:26.540665 10156 tablet_service.cc:1514] Processing DeleteTablet for tablet 49c1c2f4282b4679bad023eaf121456b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:21:26 UTC) from {username='oryx', principal='oryx/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:33026
I20260621 05:21:26.540795 10273 tablet_replica.cc:330] T 49c1c2f4282b4679bad023eaf121456b P 889e832a1c314be69ec5921b3abc3cd5: stopping tablet replica
I20260621 05:21:26.540858 10273 raft_consensus.cc:2229] T 49c1c2f4282b4679bad023eaf121456b P 889e832a1c314be69ec5921b3abc3cd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:21:26.540895 10273 raft_consensus.cc:2258] T 49c1c2f4282b4679bad023eaf121456b P 889e832a1c314be69ec5921b3abc3cd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:21:26.541219 10273 ts_tablet_manager.cc:1905] T 49c1c2f4282b4679bad023eaf121456b P 889e832a1c314be69ec5921b3abc3cd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:21:26.542550 10273 ts_tablet_manager.cc:1918] T 49c1c2f4282b4679bad023eaf121456b P 889e832a1c314be69ec5921b3abc3cd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 05:21:26.542603 10273 log.cc:1196] T 49c1c2f4282b4679bad023eaf121456b P 889e832a1c314be69ec5921b3abc3cd5: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/49c1c2f4282b4679bad023eaf121456b
I20260621 05:21:26.542852 10273 ts_tablet_manager.cc:1939] T 49c1c2f4282b4679bad023eaf121456b P 889e832a1c314be69ec5921b3abc3cd5: Deleting consensus metadata
I20260621 05:21:26.543210 9726 catalog_manager.cc:4841] TS 889e832a1c314be69ec5921b3abc3cd5 (127.2.32.3:37013): tablet 49c1c2f4282b4679bad023eaf121456b (table test-table [id=dc2c0f7400774e7b8f40bb25475912df]) successfully deleted
W20260621 05:21:26.544647 9726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.32.1:36237 (error 108)
I20260621 05:21:26.544879 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 9951
I20260621 05:21:26.551550 10347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:26.544975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:36237 (local address 127.0.0.1:41666)
0621 05:21:26.551429 (+ 6454us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:26.551498 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.1:36237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6340,"thread_start_us":45,"threads_started":1}
W20260621 05:21:26.551800 9726 catalog_manager.cc:4584] TS cfa08fd3741046fd93232e135785615c (127.2.32.1:36237): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 49c1c2f4282b4679bad023eaf121456b: Network error: Client connection negotiation failed: client connection to 127.2.32.1:36237: connect: Connection refused (error 111)
I20260621 05:21:26.553128 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 10091
I20260621 05:21:26.560483 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 9707
2026-06-21T05:21:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3046 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:21:26 dist-test-slave-b357 krb5kdc[10350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:21:26 dist-test-slave-b357 krb5kdc[10350](info): set up 2 sockets
Jun 21 05:21:26 dist-test-slave-b357 krb5kdc[10350](info): commencing operation
krb5kdc: starting...
W20260621 05:21:27.611910 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:21:27 dist-test-slave-b357 krb5kdc[10350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:21:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:21:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:21:27.742215 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:37561
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:45477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:37561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:21:27.821008 10363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:27.821234 10363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:27.821269 10363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:27.822376 10363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:21:27.822443 10363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:27.822465 10363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:27.822494 10363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:21:27.822523 10363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:21:27.824038 10363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:45477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:37561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:37561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:27.824293 10363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:27.824513 10363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:27.829761 10371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:27.829957 10363 server_base.cc:1034] running on GCE node
W20260621 05:21:27.829761 10369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:27.829756 10368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:27.830439 10363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:27.830802 10363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:27.831990 10363 hybrid_clock.cc:648] HybridClock initialized: now 1782019287831947 us; error 66 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-21T05:21:27Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1278 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:21:27 dist-test-slave-b357 krb5kdc[10376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:21:27 dist-test-slave-b357 krb5kdc[10376](info): set up 2 sockets
Jun 21 05:21:27 dist-test-slave-b357 krb5kdc[10376](info): commencing operation
krb5kdc: starting...
W20260621 05:21:28.900265 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:21:28 dist-test-slave-b357 krb5kdc[10376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:21:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:21:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:21:29.031121 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:40659
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:40659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:21:29.115980 10389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:29.116230 10389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:29.116264 10389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:29.117375 10389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:21:29.117448 10389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:29.117472 10389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:29.117499 10389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:21:29.117529 10389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:21:29.118978 10389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:40659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:40659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:29.119288 10389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:29.119514 10389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:29.124457 10394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:29.124464 10397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:29.124426 10389 server_base.cc:1034] running on GCE node
W20260621 05:21:29.124423 10395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:29.124898 10389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:29.125191 10389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:29.126387 10389 hybrid_clock.cc:648] HybridClock initialized: now 1782019289126373 us; error 53 us; skew 500 ppm
Jun 21 05:21:29 dist-test-slave-b357 krb5kdc[10376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019289, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:21:29.129626 10389 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:21:29.130154 10389 webserver.cc:460] Webserver started at http://127.2.32.62:34401/ using document root <none> and password file <none>
I20260621 05:21:29.130399 10389 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:29.130470 10389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:29.130630 10389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:21:29.131809 10389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "d34638cf71b7462a871a3ca4b22cdcce"
format_stamp: "Formatted at 2026-06-21 05:21:29 on dist-test-slave-b357"
server_key: "29c6c45e75231adfb04a2faea5875ff1"
server_key_iv: "0357a6cbe1cc20ec59acce5aa993308e"
server_key_version: "encryptionkey@0"
I20260621 05:21:29.132185 10389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "d34638cf71b7462a871a3ca4b22cdcce"
format_stamp: "Formatted at 2026-06-21 05:21:29 on dist-test-slave-b357"
server_key: "29c6c45e75231adfb04a2faea5875ff1"
server_key_iv: "0357a6cbe1cc20ec59acce5aa993308e"
server_key_version: "encryptionkey@0"
I20260621 05:21:29.133584 10389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:21:29.134583 10404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:29.134814 10389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:29.134924 10389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "d34638cf71b7462a871a3ca4b22cdcce"
format_stamp: "Formatted at 2026-06-21 05:21:29 on dist-test-slave-b357"
server_key: "29c6c45e75231adfb04a2faea5875ff1"
server_key_iv: "0357a6cbe1cc20ec59acce5aa993308e"
server_key_version: "encryptionkey@0"
I20260621 05:21:29.135012 10389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:29.168632 10389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:29.170992 10389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:29.171271 10389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:29.175511 10389 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:40659
I20260621 05:21:29.175545 10456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:40659 every 8 connection(s)
I20260621 05:21:29.175932 10389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:21:29.176782 10457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:29.177790 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 10389
I20260621 05:21:29.177925 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:21:29.178130 2176 external_mini_cluster.cc:1428] Setting key 03ecee745f0930f59a6005848fad75db
I20260621 05:21:29.179418 10457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce: Bootstrap starting.
I20260621 05:21:29.180152 10457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:29.180498 10457 log.cc:824] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce: Log is configured to *not* fsync() on all Append() calls
I20260621 05:21:29.181185 10457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce: No bootstrap required, opened a new log
I20260621 05:21:29.183722 10457 raft_consensus.cc:348] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34638cf71b7462a871a3ca4b22cdcce" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40659 } }
I20260621 05:21:29.183930 10457 raft_consensus.cc:374] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:29.184006 10457 raft_consensus.cc:729] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d34638cf71b7462a871a3ca4b22cdcce, State: Initialized, Role: FOLLOWER
I20260621 05:21:29.184279 10457 consensus_queue.cc:260] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34638cf71b7462a871a3ca4b22cdcce" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40659 } }
I20260621 05:21:29.184393 10457 raft_consensus.cc:388] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:21:29.184434 10457 raft_consensus.cc:482] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:21:29.184489 10457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:29.185217 10457 raft_consensus.cc:504] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34638cf71b7462a871a3ca4b22cdcce" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40659 } }
I20260621 05:21:29.185338 10457 leader_election.cc:302] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d34638cf71b7462a871a3ca4b22cdcce; no voters:
I20260621 05:21:29.185494 10457 leader_election.cc:288] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:21:29.185532 10462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:21:29.185669 10462 raft_consensus.cc:686] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [term 1 LEADER]: Becoming Leader. State: Replica: d34638cf71b7462a871a3ca4b22cdcce, State: Running, Role: LEADER
I20260621 05:21:29.185761 10462 consensus_queue.cc:237] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34638cf71b7462a871a3ca4b22cdcce" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40659 } }
I20260621 05:21:29.185806 10457 sys_catalog.cc:564] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:21:29.186124 10462 sys_catalog.cc:455] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d34638cf71b7462a871a3ca4b22cdcce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34638cf71b7462a871a3ca4b22cdcce" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40659 } } }
I20260621 05:21:29.186200 10462 sys_catalog.cc:458] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [sys.catalog]: This master's current role is: LEADER
I20260621 05:21:29.186316 10462 sys_catalog.cc:455] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [sys.catalog]: SysCatalogTable state changed. Reason: New leader d34638cf71b7462a871a3ca4b22cdcce. Latest consensus state: current_term: 1 leader_uuid: "d34638cf71b7462a871a3ca4b22cdcce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34638cf71b7462a871a3ca4b22cdcce" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40659 } } }
I20260621 05:21:29.186372 10462 sys_catalog.cc:458] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [sys.catalog]: This master's current role is: LEADER
Jun 21 05:21:29 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
W20260621 05:21:29.188323 10477 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:21:29.188710 10477 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:21:29.188838 10466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:21:29.190019 10466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:21:29.192137 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:29.178654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43532 (local address 127.2.32.62:40659)
0621 05:21:29.178844 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:29.178853 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:29.178883 (+ 30us) server_negotiation.cc:407] Connection header received
0621 05:21:29.179409 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:29.179415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:29.179460 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:29.179612 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:29.180440 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:29.181438 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:29.182503 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:29.183288 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:29.183425 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:29.187574 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:29.187578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:29.187578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:21:29.187593 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:29.189797 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:29.190393 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:29.190399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:29.190399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:29.190661 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:29.190976 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:29.190977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:29.190978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:29.191410 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:29.191632 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:29.191703 (+ 71us) server_negotiation.cc:299] Negotiation successful
0621 05:21:29.191855 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
I20260621 05:21:29.192787 10466 catalog_manager.cc:1347] Generated new cluster ID: 1a77e8314a904be2ae02da1890277bc6
I20260621 05:21:29.192883 10466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:21:29.204677 10466 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:21:29.205251 10466 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:21:29.215530 10466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce: Generated new TSK 0
I20260621 05:21:29.215724 10466 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
I20260621 05:21:29.246855 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:40659
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:21:29.341092 10483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:29.341321 10483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:29.341351 10483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:29.342640 10483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:29.342689 10483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:29.342720 10483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:21:29.345146 10483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:29.345523 10483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:29.345857 10483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:29.351701 10489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:29.351701 10488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:29.351763 10491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:29.352177 10483 server_base.cc:1034] running on GCE node
I20260621 05:21:29.352496 10483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:29.352846 10483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:29.354039 10483 hybrid_clock.cc:648] HybridClock initialized: now 1782019289354014 us; error 48 us; skew 500 ppm
Jun 21 05:21:29 dist-test-slave-b357 krb5kdc[10376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019289, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:21:29.357508 10483 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:21:29.358165 10483 webserver.cc:460] Webserver started at http://127.2.32.1:43779/ using document root <none> and password file <none>
I20260621 05:21:29.358433 10483 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:29.358508 10483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:29.358667 10483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:21:29.359848 10483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "a2aac315512d4769893f7eb5040ea6ec"
format_stamp: "Formatted at 2026-06-21 05:21:29 on dist-test-slave-b357"
server_key: "6fb24d79f921e48c06ece6935bd07540"
server_key_iv: "abf0d8a99c7975fdf6a9d1561136edc7"
server_key_version: "encryptionkey@0"
I20260621 05:21:29.360217 10483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "a2aac315512d4769893f7eb5040ea6ec"
format_stamp: "Formatted at 2026-06-21 05:21:29 on dist-test-slave-b357"
server_key: "6fb24d79f921e48c06ece6935bd07540"
server_key_iv: "abf0d8a99c7975fdf6a9d1561136edc7"
server_key_version: "encryptionkey@0"
I20260621 05:21:29.361685 10483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:21:29.362759 10498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:29.362993 10483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:29.363103 10483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "a2aac315512d4769893f7eb5040ea6ec"
format_stamp: "Formatted at 2026-06-21 05:21:29 on dist-test-slave-b357"
server_key: "6fb24d79f921e48c06ece6935bd07540"
server_key_iv: "abf0d8a99c7975fdf6a9d1561136edc7"
server_key_version: "encryptionkey@0"
I20260621 05:21:29.363232 10483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:29.392067 10483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:29.394853 10483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:29.395116 10483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:29.395467 10483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:21:29.395874 10483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:21:29.395931 10483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:29.395968 10483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:21:29.396025 10483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:29.402426 10483 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:44341
I20260621 05:21:29.402518 10611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:44341 every 8 connection(s)
I20260621 05:21:29.402880 10483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:21:29.403398 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 10483
I20260621 05:21:29.403498 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:21:29.403735 2176 external_mini_cluster.cc:1428] Setting key 45986753d30bcea62cc6ccb971fa5f6a
Jun 21 05:21:29 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019289, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.32.2@KRBTEST.COM; defaulting to no policy
I20260621 05:21:29.413479 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:29.403923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:51719 (local address 127.2.32.62:40659)
0621 05:21:29.403975 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:29.403977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:29.404153 (+ 176us) server_negotiation.cc:407] Connection header received
0621 05:21:29.404658 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:29.404659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:29.404672 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:29.404746 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:29.405702 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:29.406250 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:29.407493 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:29.407863 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:29.407946 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:29.410666 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:29.410667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:29.410668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:21:29.410675 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:29.412074 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:29.412502 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:29.412503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:29.412504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:29.412560 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:29.412823 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:29.412823 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:29.412824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:29.412936 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:29.413010 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:29.413299 (+ 289us) server_negotiation.cc:299] Negotiation successful
0621 05:21:29.413385 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:21:29.413674 10614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:29.403881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.1:51719)
0621 05:21:29.404035 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:29.404076 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:29.404559 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:29.404795 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:29.404798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:29.405364 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:29.405588 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:29.405594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:29.406348 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:29.406349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:29.407386 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:29.407389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:29.408036 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:29.408038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:29.408208 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:29.408976 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:29.408986 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:29.410562 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:21:29.412179 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:29.412186 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:29.412188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:29.412411 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:29.412627 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:29.412628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:29.412628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:29.412746 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:29.413000 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:21:29.413007 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:21:29.413148 (+ 141us) client_negotiation.cc:769] Sending connection context
0621 05:21:29.413270 (+ 122us) client_negotiation.cc:240] Negotiation successful
0621 05:21:29.413432 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260621 05:21:29.414232 10612 heartbeater.cc:347] Connected to a master server at 127.2.32.62:40659
I20260621 05:21:29.414312 10612 heartbeater.cc:464] Registering TS with master...
I20260621 05:21:29.414498 10612 heartbeater.cc:511] Master 127.2.32.62:40659 requested a full tablet report, sending...
I20260621 05:21:29.415289 10420 ts_manager.cc:194] Registered new tserver with Master: a2aac315512d4769893f7eb5040ea6ec (127.2.32.1:44341)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.2@KRBTEST.COM" created.
I20260621 05:21:29.416082 10420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:51719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
I20260621 05:21:29.432399 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:40659
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:21:29.515298 10617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:29.515498 10617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:29.515520 10617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:29.516589 10617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:29.516628 10617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:29.516654 10617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:21:29.518081 10617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:29.518263 10617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:29.518424 10617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:21:29.523373 10617 server_base.cc:1034] running on GCE node
W20260621 05:21:29.523346 10622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:29.523342 10623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:29.523310 10625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:29.523720 10617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:29.523962 10617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:29.525156 10617 hybrid_clock.cc:648] HybridClock initialized: now 1782019289525142 us; error 33 us; skew 500 ppm
Jun 21 05:21:29 dist-test-slave-b357 krb5kdc[10376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019289, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:21:29.527959 10617 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:21:29.528415 10617 webserver.cc:460] Webserver started at http://127.2.32.2:33209/ using document root <none> and password file <none>
I20260621 05:21:29.528658 10617 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:29.528728 10617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:29.528908 10617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:21:29.529950 10617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "6d12a35998a442f2b8c5d83fcf0e08f1"
format_stamp: "Formatted at 2026-06-21 05:21:29 on dist-test-slave-b357"
server_key: "8df1e9371ad2a4dc793779b7e0057795"
server_key_iv: "446e40484cdf025a19499c5f97e15766"
server_key_version: "encryptionkey@0"
I20260621 05:21:29.530313 10617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "6d12a35998a442f2b8c5d83fcf0e08f1"
format_stamp: "Formatted at 2026-06-21 05:21:29 on dist-test-slave-b357"
server_key: "8df1e9371ad2a4dc793779b7e0057795"
server_key_iv: "446e40484cdf025a19499c5f97e15766"
server_key_version: "encryptionkey@0"
I20260621 05:21:29.531787 10617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:21:29.532850 10632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:29.533238 10617 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:21:29.533358 10617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "6d12a35998a442f2b8c5d83fcf0e08f1"
format_stamp: "Formatted at 2026-06-21 05:21:29 on dist-test-slave-b357"
server_key: "8df1e9371ad2a4dc793779b7e0057795"
server_key_iv: "446e40484cdf025a19499c5f97e15766"
server_key_version: "encryptionkey@0"
I20260621 05:21:29.533434 10617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:29.544088 10617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:29.546103 10617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:29.546285 10617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:29.546519 10617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:21:29.546821 10617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:21:29.546885 10617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:29.546926 10617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:21:29.546953 10617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:29.553653 10617 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:39945
I20260621 05:21:29.553704 10745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:39945 every 8 connection(s)
I20260621 05:21:29.553990 10617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:21:29.558156 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 10617
I20260621 05:21:29.558254 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:21:29.558445 2176 external_mini_cluster.cc:1428] Setting key a7dbc31d30f88ef6531d539dca2f5dbf
Jun 21 05:21:29 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019289, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:21:29.564673 10748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:29.554774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.2:50229)
0621 05:21:29.555089 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:29.555269 (+ 180us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:29.555752 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:29.555988 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:29.555991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:29.556530 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:29.556706 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:29.556710 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:29.557425 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:29.557426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:29.558269 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:29.558272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:29.558778 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:29.558780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:29.558937 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:29.559516 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:29.559524 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:29.560985 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:21:29.563322 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:29.563328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:29.563329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:29.563546 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:29.563781 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:29.563781 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:29.563782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:29.563892 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:29.564200 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:21:29.564204 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:21:29.564315 (+ 111us) client_negotiation.cc:769] Sending connection context
0621 05:21:29.564400 (+ 85us) client_negotiation.cc:240] Negotiation successful
0621 05:21:29.564515 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":115,"threads_started":1}
I20260621 05:21:29.564679 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:29.554759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:50229 (local address 127.2.32.62:40659)
0621 05:21:29.554841 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:29.554843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:29.555390 (+ 547us) server_negotiation.cc:407] Connection header received
0621 05:21:29.555853 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:29.555854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:29.555864 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:29.555929 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:29.556829 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:29.557316 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:29.558370 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:29.558673 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:29.558803 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:29.561079 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:29.561080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:29.561080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:21:29.561089 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:29.563233 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:29.563632 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:29.563634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:29.563634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:29.563702 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:29.563963 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:29.563964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:29.563965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:29.564117 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:29.564206 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:29.564455 (+ 249us) server_negotiation.cc:299] Negotiation successful
0621 05:21:29.564549 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 05:21:29.565481 10746 heartbeater.cc:347] Connected to a master server at 127.2.32.62:40659
I20260621 05:21:29.565601 10746 heartbeater.cc:464] Registering TS with master...
I20260621 05:21:29.565828 10746 heartbeater.cc:511] Master 127.2.32.62:40659 requested a full tablet report, sending...
I20260621 05:21:29.566277 10420 ts_manager.cc:194] Registered new tserver with Master: 6d12a35998a442f2b8c5d83fcf0e08f1 (127.2.32.2:39945)
I20260621 05:21:29.566634 10420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:50229
WARNING: no policy specified for kudu/127.2.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
I20260621 05:21:29.587810 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:40659
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:21:29.669514 10751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:29.669710 10751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:29.669734 10751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:29.670794 10751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:29.670828 10751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:29.670853 10751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:21:29.672188 10751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:29.672457 10751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:29.672701 10751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:29.678373 10756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:29.678325 10759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:29.678356 10751 server_base.cc:1034] running on GCE node
W20260621 05:21:29.678321 10757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:29.678782 10751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:29.679013 10751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:29.680210 10751 hybrid_clock.cc:648] HybridClock initialized: now 1782019289680207 us; error 42 us; skew 500 ppm
Jun 21 05:21:29 dist-test-slave-b357 krb5kdc[10376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019289, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:21:29.682793 10751 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:21:29.683358 10751 webserver.cc:460] Webserver started at http://127.2.32.3:45063/ using document root <none> and password file <none>
I20260621 05:21:29.683621 10751 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:29.683697 10751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:29.683861 10751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:21:29.684954 10751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "0750a6312cb44003a717862616a1bab8"
format_stamp: "Formatted at 2026-06-21 05:21:29 on dist-test-slave-b357"
server_key: "22e72891c85d5407a878db3962049fbf"
server_key_iv: "36e889385a808df6a99eea78ff53e191"
server_key_version: "encryptionkey@0"
I20260621 05:21:29.685375 10751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "0750a6312cb44003a717862616a1bab8"
format_stamp: "Formatted at 2026-06-21 05:21:29 on dist-test-slave-b357"
server_key: "22e72891c85d5407a878db3962049fbf"
server_key_iv: "36e889385a808df6a99eea78ff53e191"
server_key_version: "encryptionkey@0"
I20260621 05:21:29.687248 10751 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260621 05:21:29.688643 10766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:29.688946 10751 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:29.689091 10751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "0750a6312cb44003a717862616a1bab8"
format_stamp: "Formatted at 2026-06-21 05:21:29 on dist-test-slave-b357"
server_key: "22e72891c85d5407a878db3962049fbf"
server_key_iv: "36e889385a808df6a99eea78ff53e191"
server_key_version: "encryptionkey@0"
I20260621 05:21:29.689182 10751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:29.700112 10751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:29.702389 10751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:29.702569 10751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:29.702834 10751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:21:29.703204 10751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:21:29.703261 10751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:29.703307 10751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:21:29.703337 10751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:29.709518 10751 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:36349
I20260621 05:21:29.709618 10879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:36349 every 8 connection(s)
I20260621 05:21:29.709894 10751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:21:29.713482 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 10751
I20260621 05:21:29.713626 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:21:29.713822 2176 external_mini_cluster.cc:1428] Setting key 08cd02bbe2777e2d8252f113482eb595
Jun 21 05:21:29 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019289, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:21:29.720703 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:29.710594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:59415 (local address 127.2.32.62:40659)
0621 05:21:29.710672 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:29.710674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:29.710818 (+ 144us) server_negotiation.cc:407] Connection header received
0621 05:21:29.711427 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:29.711429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:29.711446 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:29.711540 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:29.712448 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:29.713025 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:29.714779 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:29.715037 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:29.715180 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:29.717847 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:29.717849 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:29.717850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:21:29.717859 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:29.719329 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:29.719782 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:29.719783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:29.719783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:29.719839 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:29.720093 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:29.720093 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:29.720094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:29.720219 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:29.720277 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:29.720552 (+ 275us) server_negotiation.cc:299] Negotiation successful
0621 05:21:29.720619 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 05:21:29.720872 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:29.710555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.3:59415)
0621 05:21:29.710705 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:29.710738 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:29.711257 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:29.711594 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:29.711597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:29.712153 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:29.712335 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:29.712340 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:29.713177 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:29.713179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:29.714662 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:29.714666 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:29.715180 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:29.715181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:29.715361 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:29.716061 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:29.716072 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:29.717723 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:21:29.719425 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:29.719431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:29.719433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:29.719681 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:29.719915 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:29.719916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:29.719916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:29.720010 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:29.720301 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:21:29.720307 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:21:29.720429 (+ 122us) client_negotiation.cc:769] Sending connection context
0621 05:21:29.720525 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 05:21:29.720652 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260621 05:21:29.721506 10880 heartbeater.cc:347] Connected to a master server at 127.2.32.62:40659
I20260621 05:21:29.721603 10880 heartbeater.cc:464] Registering TS with master...
I20260621 05:21:29.721740 10880 heartbeater.cc:511] Master 127.2.32.62:40659 requested a full tablet report, sending...
I20260621 05:21:29.722139 10420 ts_manager.cc:194] Registered new tserver with Master: 0750a6312cb44003a717862616a1bab8 (127.2.32.3:36349)
I20260621 05:21:29.722477 10420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:59415
I20260621 05:21:29.725747 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.1.keytab.
Entry for principal oryx/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.1.keytab.
WARNING: no policy specified for oryx/127.2.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.2.keytab.
Entry for principal oryx/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.2.keytab.
WARNING: no policy specified for oryx/127.2.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.3.keytab.
Entry for principal oryx/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.3.keytab.
I20260621 05:21:29.802284 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 10483
I20260621 05:21:29.806691 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 10617
I20260621 05:21:29.811232 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 10751
I20260621 05:21:29.815510 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 10389
I20260621 05:21:29.822165 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:40659
--webserver_interface=127.2.32.62
--webserver_port=34401
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:40659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:21:29.903723 10889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:29.903916 10889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:29.903939 10889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:29.904981 10889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:21:29.905017 10889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:29.905031 10889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:29.905050 10889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:21:29.905063 10889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:21:29.906401 10889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:40659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:40659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=34401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:29.906579 10889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:29.906738 10889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:29.911690 10894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:29.911698 10895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:29.911842 10889 server_base.cc:1034] running on GCE node
W20260621 05:21:29.911698 10897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:29.912388 10889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:29.912734 10889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:29.913962 10889 hybrid_clock.cc:648] HybridClock initialized: now 1782019289913944 us; error 42 us; skew 500 ppm
Jun 21 05:21:29 dist-test-slave-b357 krb5kdc[10376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019289, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:21:29.916849 10889 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:21:29.917317 10889 webserver.cc:460] Webserver started at http://127.2.32.62:34401/ using document root <none> and password file <none>
I20260621 05:21:29.917544 10889 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:29.917625 10889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:29.919293 10889 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:29.920006 10904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:29.920430 10889 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:29.920565 10889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "d34638cf71b7462a871a3ca4b22cdcce"
format_stamp: "Formatted at 2026-06-21 05:21:29 on dist-test-slave-b357"
server_key: "29c6c45e75231adfb04a2faea5875ff1"
server_key_iv: "0357a6cbe1cc20ec59acce5aa993308e"
server_key_version: "encryptionkey@0"
I20260621 05:21:29.920953 10889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:29.927551 10889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:29.930305 10889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:29.930562 10889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:29.934828 10889 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:40659
I20260621 05:21:29.934867 10956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:40659 every 8 connection(s)
I20260621 05:21:29.935340 10889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:21:29.936735 10957 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:21:29.937390 10957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce: Bootstrap starting.
I20260621 05:21:29.937450 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 10889
I20260621 05:21:29.937752 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:44341
--local_ip_for_outbound_sockets=127.2.32.1
--tserver_master_addrs=127.2.32.62:40659
--webserver_port=43779
--webserver_interface=127.2.32.1
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.32.1
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:21:29.945761 10957 log.cc:824] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce: Log is configured to *not* fsync() on all Append() calls
I20260621 05:21:29.947479 10957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:21:29.947757 10957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce: Bootstrap complete.
I20260621 05:21:29.949954 10957 raft_consensus.cc:348] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34638cf71b7462a871a3ca4b22cdcce" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40659 } }
I20260621 05:21:29.950191 10957 raft_consensus.cc:729] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d34638cf71b7462a871a3ca4b22cdcce, State: Initialized, Role: FOLLOWER
I20260621 05:21:29.950392 10957 consensus_queue.cc:260] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34638cf71b7462a871a3ca4b22cdcce" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40659 } }
I20260621 05:21:29.950487 10957 raft_consensus.cc:388] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:21:29.950551 10957 raft_consensus.cc:482] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:21:29.950598 10957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [term 1 FOLLOWER]: Advancing to term 2
I20260621 05:21:29.951613 10957 raft_consensus.cc:504] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34638cf71b7462a871a3ca4b22cdcce" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40659 } }
I20260621 05:21:29.951730 10957 leader_election.cc:302] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d34638cf71b7462a871a3ca4b22cdcce; no voters:
I20260621 05:21:29.951922 10957 leader_election.cc:288] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 05:21:29.951963 10961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [term 2 FOLLOWER]: Leader election won for term 2
I20260621 05:21:29.952090 10961 raft_consensus.cc:686] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [term 2 LEADER]: Becoming Leader. State: Replica: d34638cf71b7462a871a3ca4b22cdcce, State: Running, Role: LEADER
I20260621 05:21:29.952203 10957 sys_catalog.cc:564] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:21:29.952180 10961 consensus_queue.cc:237] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34638cf71b7462a871a3ca4b22cdcce" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40659 } }
I20260621 05:21:29.952456 10963 sys_catalog.cc:455] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d34638cf71b7462a871a3ca4b22cdcce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34638cf71b7462a871a3ca4b22cdcce" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40659 } } }
I20260621 05:21:29.952548 10963 sys_catalog.cc:458] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [sys.catalog]: This master's current role is: LEADER
I20260621 05:21:29.952451 10961 sys_catalog.cc:455] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [sys.catalog]: SysCatalogTable state changed. Reason: New leader d34638cf71b7462a871a3ca4b22cdcce. Latest consensus state: current_term: 2 leader_uuid: "d34638cf71b7462a871a3ca4b22cdcce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34638cf71b7462a871a3ca4b22cdcce" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40659 } } }
I20260621 05:21:29.952732 10961 sys_catalog.cc:458] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce [sys.catalog]: This master's current role is: LEADER
I20260621 05:21:29.953105 10966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:21:29.953688 10966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:21:29.954013 10966 catalog_manager.cc:1259] Loaded cluster ID: 1a77e8314a904be2ae02da1890277bc6
I20260621 05:21:29.954063 10966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:21:29.955930 10966 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:21:29.956121 10966 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d34638cf71b7462a871a3ca4b22cdcce: Loaded TSK: 0
I20260621 05:21:29.956334 10966 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 05:21:30.019770 10959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:30.019963 10959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:30.019985 10959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:30.021117 10959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:30.021157 10959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:30.021183 10959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:21:30.022629 10959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.1.keytab
--principal=oryx/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:44341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=43779
--tserver_master_addrs=127.2.32.62:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:30.022838 10959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:30.023074 10959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:30.028347 10983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:30.028393 10986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:30.028618 10984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:30.028738 10959 server_base.cc:1034] running on GCE node
I20260621 05:21:30.028971 10959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:30.029237 10959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:30.030463 10959 hybrid_clock.cc:648] HybridClock initialized: now 1782019290030424 us; error 40 us; skew 500 ppm
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019290, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:21:30.033928 10959 init.cc:377] Logged in from keytab as oryx/127.2.32.1@KRBTEST.COM (short username oryx)
I20260621 05:21:30.034636 10959 webserver.cc:460] Webserver started at http://127.2.32.1:43779/ using document root <none> and password file <none>
I20260621 05:21:30.034907 10959 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:30.035004 10959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:30.036810 10959 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:30.037639 10993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:30.037822 10959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:30.037904 10959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "a2aac315512d4769893f7eb5040ea6ec"
format_stamp: "Formatted at 2026-06-21 05:21:29 on dist-test-slave-b357"
server_key: "6fb24d79f921e48c06ece6935bd07540"
server_key_iv: "abf0d8a99c7975fdf6a9d1561136edc7"
server_key_version: "encryptionkey@0"
I20260621 05:21:30.038288 10959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:30.059322 10959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:30.061741 10959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:30.061928 10959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:30.062218 10959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:21:30.062568 10959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:21:30.062606 10959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:30.062633 10959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:21:30.062646 10959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:30.071072 10959 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:44341
I20260621 05:21:30.071677 10959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:21:30.072160 11106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:44341 every 8 connection(s)
I20260621 05:21:30.073171 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 10959
I20260621 05:21:30.073395 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:39945
--local_ip_for_outbound_sockets=127.2.32.2
--tserver_master_addrs=127.2.32.62:40659
--webserver_port=33209
--webserver_interface=127.2.32.2
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.32.2
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 05:21:30.083626 11110 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:30.072547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:56359 (local address 127.2.32.62:40659)
0621 05:21:30.072723 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:30.072732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:30.072776 (+ 44us) server_negotiation.cc:407] Connection header received
0621 05:21:30.073876 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:30.073882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:30.073933 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:30.074047 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:30.075373 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.076920 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.078574 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.079263 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.079373 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:30.083393 (+ 4020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.1:56359: BlockingRecv error: recv got EOF from 127.2.32.1:56359 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
I20260621 05:21:30.084049 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:30.072442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.1:56359)
0621 05:21:30.072629 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:30.072675 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:30.073419 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:30.074178 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:30.074182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:30.074963 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:30.075245 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:30.075251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.077050 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.077052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:30.078453 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:30.078456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.079472 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.079473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:30.079635 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:30.080246 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:30.080257 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:30.083838 (+ 3581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
W20260621 05:21:30.084275 11107 heartbeater.cc:644] Failed to heartbeat to 127.2.32.62:40659 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.32.62:40659: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:30.089849 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:30.084507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.1:50073)
0621 05:21:30.084644 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:30.084656 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:30.084717 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:30.084887 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:30.084888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:30.085048 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:30.085100 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:30.085101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.085705 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.085706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:30.086547 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:30.086549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.087099 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.087101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:30.087238 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:30.087773 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:30.087777 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:30.089772 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
W20260621 05:21:30.090178 11110 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:30.084636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:50073 (local address 127.2.32.62:40659)
0621 05:21:30.084672 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:30.084674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:30.084773 (+ 99us) server_negotiation.cc:407] Connection header received
0621 05:21:30.084796 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:30.084796 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:30.084808 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:30.084861 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:30.085187 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.085633 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.086644 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.086968 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.087035 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:30.090092 (+ 3057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.1:50073: BlockingRecv error: recv got EOF from 127.2.32.1:50073 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:30.095700 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:30.090066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.1:60685)
0621 05:21:30.090316 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:30.090326 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:30.090383 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:30.090567 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:30.090568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:30.090724 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:30.090758 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:30.090760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.091547 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.091548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:30.092363 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:30.092365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.092876 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.092877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:30.092929 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:30.093443 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:30.093448 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:30.095596 (+ 2148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238}
W20260621 05:21:30.095777 11110 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:30.090147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:60685 (local address 127.2.32.62:40659)
0621 05:21:30.090322 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:30.090325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:30.090440 (+ 115us) server_negotiation.cc:407] Connection header received
0621 05:21:30.090470 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:30.090471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:30.090482 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:30.090539 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:30.090882 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.091461 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.092497 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.092729 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.092864 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:30.095690 (+ 2826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.1:60685: BlockingRecv error: recv got EOF from 127.2.32.1:60685 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":116,"threads_started":1}
W20260621 05:21:30.095892 11107 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 05:21:30.160012 11111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:30.160218 11111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:30.160243 11111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:30.161306 11111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:30.161342 11111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:30.161368 11111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:21:30.162685 11111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.2.keytab
--principal=oryx/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:39945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=33209
--tserver_master_addrs=127.2.32.62:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:30.162856 11111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:30.163020 11111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:30.167920 11121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:30.167974 11118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:30.167927 11119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:30.168229 11111 server_base.cc:1034] running on GCE node
I20260621 05:21:30.168433 11111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:30.168671 11111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:30.169893 11111 hybrid_clock.cc:648] HybridClock initialized: now 1782019290169868 us; error 65 us; skew 500 ppm
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019290, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:21:30.173146 11111 init.cc:377] Logged in from keytab as oryx/127.2.32.2@KRBTEST.COM (short username oryx)
I20260621 05:21:30.173749 11111 webserver.cc:460] Webserver started at http://127.2.32.2:33209/ using document root <none> and password file <none>
I20260621 05:21:30.174055 11111 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:30.174170 11111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:30.175966 11111 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:21:30.177004 11128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:30.177388 11111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:30.177498 11111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "6d12a35998a442f2b8c5d83fcf0e08f1"
format_stamp: "Formatted at 2026-06-21 05:21:29 on dist-test-slave-b357"
server_key: "8df1e9371ad2a4dc793779b7e0057795"
server_key_iv: "446e40484cdf025a19499c5f97e15766"
server_key_version: "encryptionkey@0"
I20260621 05:21:30.177907 11111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:30.193593 11111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:30.195935 11111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:30.196100 11111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:30.196341 11111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:21:30.196645 11111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:21:30.196697 11111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:30.196740 11111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:21:30.196768 11111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:30.202651 11111 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:39945
I20260621 05:21:30.202708 11241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:39945 every 8 connection(s)
I20260621 05:21:30.203069 11111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:21:30.208889 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 11111
I20260621 05:21:30.209226 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:36349
--local_ip_for_outbound_sockets=127.2.32.3
--tserver_master_addrs=127.2.32.62:40659
--webserver_port=45063
--webserver_interface=127.2.32.3
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.32.3
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:30.212157 11244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:30.203788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.2:50277)
0621 05:21:30.204020 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:30.204081 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:30.204629 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:30.204909 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:30.204912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:30.205522 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:30.205731 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:30.205735 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.206522 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.206523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:30.207467 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:30.207469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.207889 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.207891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:30.208018 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:30.208714 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:30.208727 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:30.211978 (+ 3251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":83,"threads_started":1}
W20260621 05:21:30.212221 11110 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:30.203860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:50277 (local address 127.2.32.62:40659)
0621 05:21:30.203984 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:30.203988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:30.204186 (+ 198us) server_negotiation.cc:407] Connection header received
0621 05:21:30.204756 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:30.204757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:30.204774 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:30.204831 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:30.205838 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.206408 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.207570 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.207802 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.207884 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:30.212125 (+ 4241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.2:50277: BlockingRecv error: recv got EOF from 127.2.32.2:50277 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260621 05:21:30.212471 11242 heartbeater.cc:644] Failed to heartbeat to 127.2.32.62:40659 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.32.62:40659: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:30.217862 11244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:30.212742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.2:42245)
0621 05:21:30.212779 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:30.212791 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:30.212865 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:30.213017 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:30.213017 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:30.213152 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:30.213243 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:30.213245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.213988 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.213989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:30.214535 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:30.214536 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.214903 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.214903 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:30.214969 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:30.215475 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:30.215480 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:30.217770 (+ 2290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16}
W20260621 05:21:30.217928 11110 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:30.212796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:42245 (local address 127.2.32.62:40659)
0621 05:21:30.212865 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:30.212868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:30.212907 (+ 39us) server_negotiation.cc:407] Connection header received
0621 05:21:30.212936 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:30.212937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:30.212951 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:30.213012 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:30.213330 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.213839 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.214617 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.214811 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.214904 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:30.217861 (+ 2957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.2:42245: BlockingRecv error: recv got EOF from 127.2.32.2:42245 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:30.223465 11244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:30.218613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.2:52759)
0621 05:21:30.218672 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:30.218684 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:30.218727 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:30.218901 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:30.218902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:30.219009 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:30.219046 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:30.219048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.220040 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.220041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:30.220546 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:30.220547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.221068 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.221069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:30.221112 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:30.221647 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:30.221652 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:30.223388 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260621 05:21:30.223515 11110 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:30.218717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:52759 (local address 127.2.32.62:40659)
0621 05:21:30.218760 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:30.218762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:30.218772 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:21:30.218796 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:30.218797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:30.218808 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:30.218973 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:30.219122 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.219941 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.220635 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.220955 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.221031 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:30.223453 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.2:52759: BlockingRecv error: recv got EOF from 127.2.32.2:52759 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260621 05:21:30.223956 11242 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 05:21:30.293177 11245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:30.293390 11245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:30.293413 11245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:30.294612 11245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:30.294653 11245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:30.294679 11245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:21:30.296084 11245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/krb5kdc/oryx_127.2.32.3.keytab
--principal=oryx/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:36349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=45063
--tserver_master_addrs=127.2.32.62:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:30.296309 11245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:30.296566 11245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:30.301858 11253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:30.301919 11251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:30.301919 11250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:30.302119 11245 server_base.cc:1034] running on GCE node
I20260621 05:21:30.302331 11245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:30.302526 11245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:30.303714 11245 hybrid_clock.cc:648] HybridClock initialized: now 1782019290303702 us; error 29 us; skew 500 ppm
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019290, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:21:30.306988 11245 init.cc:377] Logged in from keytab as oryx/127.2.32.3@KRBTEST.COM (short username oryx)
I20260621 05:21:30.307685 11245 webserver.cc:460] Webserver started at http://127.2.32.3:45063/ using document root <none> and password file <none>
I20260621 05:21:30.307984 11245 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:30.308061 11245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:30.309834 11245 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:21:30.310721 11260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:30.311111 11245 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:21:30.311300 11245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "0750a6312cb44003a717862616a1bab8"
format_stamp: "Formatted at 2026-06-21 05:21:29 on dist-test-slave-b357"
server_key: "22e72891c85d5407a878db3962049fbf"
server_key_iv: "36e889385a808df6a99eea78ff53e191"
server_key_version: "encryptionkey@0"
I20260621 05:21:30.311740 11245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:30.328354 11245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:30.331213 11245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:30.331426 11245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:30.331748 11245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:21:30.332154 11245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:21:30.332191 11245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:30.332219 11245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:21:30.332232 11245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:30.340432 11245 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:36349
I20260621 05:21:30.340595 11373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:36349 every 8 connection(s)
I20260621 05:21:30.340955 11245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:21:30.345271 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 11245
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:30.349541 11376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:30.341782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.3:43953)
0621 05:21:30.341970 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:30.342014 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:30.342505 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:30.342777 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:30.342781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:30.343431 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:30.343620 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:30.343624 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.344354 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.344355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:30.345249 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:30.345251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.345717 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.345718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:30.345880 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:30.346448 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:30.346456 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:30.349295 (+ 2839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
W20260621 05:21:30.349627 11110 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:30.341789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:43953 (local address 127.2.32.62:40659)
0621 05:21:30.341870 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:30.341873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:30.342111 (+ 238us) server_negotiation.cc:407] Connection header received
0621 05:21:30.342615 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:30.342616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:30.342627 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:30.342695 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:30.343729 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.344241 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.345357 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.345609 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.345707 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:30.349538 (+ 3831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.3:43953: BlockingRecv error: recv got EOF from 127.2.32.3:43953 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260621 05:21:30.349820 11374 heartbeater.cc:644] Failed to heartbeat to 127.2.32.62:40659 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.32.62:40659: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:30.355037 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:30.345855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43542 (local address 127.2.32.62:40659)
0621 05:21:30.345910 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:30.345911 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:30.345979 (+ 68us) server_negotiation.cc:407] Connection header received
0621 05:21:30.346053 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:30.346053 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:30.346063 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:30.346148 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:30.346819 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.347655 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.349056 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.349362 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.349455 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:30.351354 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:30.351358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:30.351358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:21:30.351367 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:30.353437 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:30.353953 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:30.353956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:30.353957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:30.354067 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:30.354306 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:30.354306 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:30.354307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:30.354539 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:30.354683 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:30.354811 (+ 128us) server_negotiation.cc:299] Negotiation successful
0621 05:21:30.354925 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:30.355856 11376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:30.350129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.3:44659)
0621 05:21:30.350265 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:30.350295 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:30.350348 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:30.350529 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:30.350530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:30.350672 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:30.350737 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:30.350739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.351639 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.351641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:30.352590 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:30.352594 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.352969 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.352970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:30.353036 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:30.353549 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:30.353554 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:30.355764 (+ 2210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112}
W20260621 05:21:30.355986 11110 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:30.350196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:44659 (local address 127.2.32.62:40659)
0621 05:21:30.350249 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:30.350251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:30.350357 (+ 106us) server_negotiation.cc:407] Connection header received
0621 05:21:30.350408 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:30.350409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:30.350421 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:30.350496 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:30.350838 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.351523 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.352685 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.352875 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.352967 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:30.355923 (+ 2956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.3:44659: BlockingRecv error: recv got EOF from 127.2.32.3:44659 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:30 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:30.360818 11376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:30.356143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.3:50775)
0621 05:21:30.356266 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:30.356278 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:30.356332 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:30.356457 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:30.356457 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:30.356570 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:30.356606 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:30.356607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.357271 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.357272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:30.357774 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:30.357776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.358181 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.358182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:30.358220 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:30.358586 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:30.358591 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:30.360719 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104}
W20260621 05:21:30.360874 11110 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:30.356232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:50775 (local address 127.2.32.62:40659)
0621 05:21:30.356272 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:30.356274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:30.356354 (+ 80us) server_negotiation.cc:407] Connection header received
0621 05:21:30.356381 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:30.356382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:30.356391 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:30.356451 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:30.356731 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.357162 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.357939 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:30.358113 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:30.358167 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:30.360805 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.3:50775: BlockingRecv error: recv got EOF from 127.2.32.3:50775 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260621 05:21:30.361066 11374 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 21 05:21:31 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:31 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:31.102223 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:31.096388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.1:49195)
0621 05:21:31.096559 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:31.096570 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:31.096640 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:31.096794 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:31.096795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:31.096954 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:31.097003 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:31.097005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:31.098135 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:31.098137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:31.098690 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:31.098692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:31.099519 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:31.099520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:31.099567 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:31.100149 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:31.100155 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:31.102147 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":110,"threads_started":1}
W20260621 05:21:31.102376 11379 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:31.096512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:49195 (local address 127.2.32.62:40659)
0621 05:21:31.096615 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:31.096617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:31.096635 (+ 18us) server_negotiation.cc:407] Connection header received
0621 05:21:31.096699 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:31.096700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:31.096713 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:31.096764 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:31.097142 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:31.097734 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:31.098819 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:31.099395 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:31.099465 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:31.102284 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.1:49195: BlockingRecv error: recv got EOF from 127.2.32.1:49195 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":54,"threads_started":1}
Jun 21 05:21:31 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:31 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:31.230051 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:31.224531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.2:55475)
0621 05:21:31.224698 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:31.224710 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:31.224773 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:31.224996 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:31.224997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:31.225229 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:31.225294 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:31.225296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:31.226074 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:31.226075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:31.226618 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:31.226620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:31.227037 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:31.227039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:31.227119 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:31.227800 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:31.227805 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:31.229961 (+ 2156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
W20260621 05:21:31.230132 11379 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:31.224638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:55475 (local address 127.2.32.62:40659)
0621 05:21:31.224703 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:31.224706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:31.224837 (+ 131us) server_negotiation.cc:407] Connection header received
0621 05:21:31.224861 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:31.224862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:31.224876 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:31.224946 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:31.225479 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:31.226003 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:31.226700 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:31.226900 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:31.226960 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:31.230080 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.2:55475: BlockingRecv error: recv got EOF from 127.2.32.2:55475 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 21 05:21:31 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:31 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:31.366617 11381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:31.361564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.3:47233)
0621 05:21:31.361794 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:31.361815 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:31.361871 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:31.362065 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:31.362066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:31.362230 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:31.362282 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:31.362283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:31.363035 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:31.363036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:31.363609 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:31.363610 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:31.363943 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:31.363944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:31.363984 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:31.364450 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:31.364454 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:31.366510 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":178,"threads_started":1}
W20260621 05:21:31.366792 11379 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:31.361746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:47233 (local address 127.2.32.62:40659)
0621 05:21:31.361891 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:31.361894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:31.361909 (+ 15us) server_negotiation.cc:407] Connection header received
0621 05:21:31.361942 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:31.361944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:31.361959 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:31.362046 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:31.362410 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:31.362912 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:31.363700 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:31.363875 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:31.363931 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:31.366711 (+ 2780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.3:47233: BlockingRecv error: recv got EOF from 127.2.32.3:47233 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jun 21 05:21:32 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:32 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:32.108863 11382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:32.102865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.1:58015)
0621 05:21:32.103016 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:32.103037 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:32.103090 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:32.103492 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:32.103494 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:32.103656 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:32.103711 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:32.103713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:32.104781 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:32.104783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:32.105378 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:32.105380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:32.105769 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:32.105770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:32.105823 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:32.106317 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:32.106322 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:32.108760 (+ 2438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
W20260621 05:21:32.108934 11383 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:32.103058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:58015 (local address 127.2.32.62:40659)
0621 05:21:32.103278 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:32.103281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:32.103296 (+ 15us) server_negotiation.cc:407] Connection header received
0621 05:21:32.103330 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:32.103331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:32.103343 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:32.103451 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:32.103888 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:32.104666 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:32.105475 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:32.105690 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:32.105770 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:32.108853 (+ 3083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.1:58015: BlockingRecv error: recv got EOF from 127.2.32.1:58015 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":141,"threads_started":1}
Jun 21 05:21:32 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:32 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:32.237145 11384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:32.230681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.2:53863)
0621 05:21:32.230870 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:32.230885 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:32.230973 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:32.231311 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:32.231313 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:32.231537 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:32.231605 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:32.231607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:32.232420 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:32.232422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:32.233104 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:32.233106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:32.233532 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:32.233533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:32.233581 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:32.234274 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:32.234279 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:32.237082 (+ 2803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":62,"threads_started":1}
W20260621 05:21:32.237353 11383 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:32.230915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:53863 (local address 127.2.32.62:40659)
0621 05:21:32.231043 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:32.231046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:32.231056 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:21:32.231174 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:32.231175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:32.231196 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:32.231266 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:32.231699 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:32.232318 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:32.233205 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:32.233436 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:32.233514 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:32.237279 (+ 3765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.2:53863: BlockingRecv error: recv got EOF from 127.2.32.2:53863 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 21 05:21:32 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:32 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:32.373548 11385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:32.367278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.3:48405)
0621 05:21:32.367445 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:32.367460 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:32.367537 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:32.367713 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:32.367714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:32.367960 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:32.368032 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:32.368035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:32.368894 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:32.368895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:32.369570 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:32.369573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:32.370130 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:32.370131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:32.370196 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:32.370914 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:32.370920 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:32.373462 (+ 2542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":60,"threads_started":1}
W20260621 05:21:32.373649 11383 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:32.367452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:48405 (local address 127.2.32.62:40659)
0621 05:21:32.367526 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:32.367528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:32.367547 (+ 19us) server_negotiation.cc:407] Connection header received
0621 05:21:32.367589 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:32.367590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:32.367604 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:32.367654 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:32.368163 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:32.368769 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:32.369695 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:32.370028 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:32.370107 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:32.373592 (+ 3485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.3:48405: BlockingRecv error: recv got EOF from 127.2.32.3:48405 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 21 05:21:33 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:33 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:33.120380 11386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:33.109443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.1:56673)
0621 05:21:33.109656 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:33.109683 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:33.109743 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:33.110069 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:33.110070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:33.110348 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:33.110411 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:33.110419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:33.111400 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:33.111402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:33.112136 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:33.112139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:33.112554 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:33.112555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:33.112607 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:33.118248 (+ 5641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:33.118253 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:33.120294 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":69,"threads_started":1}
W20260621 05:21:33.120479 11387 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:33.109510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:56673 (local address 127.2.32.62:40659)
0621 05:21:33.109708 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:33.109710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:33.109732 (+ 22us) server_negotiation.cc:407] Connection header received
0621 05:21:33.109862 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:33.109863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:33.109876 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:33.109968 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:33.110559 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:33.111290 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:33.112255 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:33.112457 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:33.112522 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:33.120406 (+ 7884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.1:56673: BlockingRecv error: recv got EOF from 127.2.32.1:56673 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":118,"threads_started":1}
Jun 21 05:21:33 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:33 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:33.243319 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:33.237716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.2:60993)
0621 05:21:33.237860 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:33.237871 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:33.237927 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:33.238202 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:33.238203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:33.238363 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:33.238417 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:33.238419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:33.239363 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:33.239363 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:33.239926 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:33.239928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:33.240391 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:33.240392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:33.240459 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:33.240984 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:33.240988 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:33.243225 (+ 2237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
W20260621 05:21:33.243346 11387 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:33.237833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:60993 (local address 127.2.32.62:40659)
0621 05:21:33.238012 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:33.238014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:33.238030 (+ 16us) server_negotiation.cc:407] Connection header received
0621 05:21:33.238068 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:33.238069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:33.238094 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:33.238174 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:33.238534 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:33.239270 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:33.240045 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:33.240264 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:33.240391 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:33.243299 (+ 2908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.2:60993: BlockingRecv error: recv got EOF from 127.2.32.2:60993 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 21 05:21:33 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:33 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:33.380319 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:33.374344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.3:54403)
0621 05:21:33.374515 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:33.374534 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:33.374598 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:33.374856 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:33.374857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:33.375068 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:33.375236 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:33.375239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:33.376144 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:33.376145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:33.376875 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:33.376881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:33.377344 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:33.377345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:33.377402 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:33.377931 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:33.377936 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:33.380235 (+ 2299us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":104,"threads_started":1}
W20260621 05:21:33.380383 11387 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:33.374521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:54403 (local address 127.2.32.62:40659)
0621 05:21:33.374591 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:33.374593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:33.374606 (+ 13us) server_negotiation.cc:407] Connection header received
0621 05:21:33.374655 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:33.374655 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:33.374673 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:33.374810 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:33.375337 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:33.376015 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:33.377023 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:33.377249 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:33.377341 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:33.380307 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.3:54403: BlockingRecv error: recv got EOF from 127.2.32.3:54403 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 21 05:21:34 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:34 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:34.125726 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:34.120903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.1:59259)
0621 05:21:34.121059 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:34.121070 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:34.121129 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:34.121322 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:34.121323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:34.121477 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:34.121526 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:34.121527 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:34.122198 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:34.122199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:34.122754 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:34.122756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:34.123161 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:34.123162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:34.123218 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:34.123775 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:34.123780 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:34.125667 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
W20260621 05:21:34.126063 11391 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:34.121046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:59259 (local address 127.2.32.62:40659)
0621 05:21:34.121183 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:34.121185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:34.121193 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:21:34.121212 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:34.121213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:34.121223 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:34.121281 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:34.121599 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:34.122105 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:34.122853 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:34.123042 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:34.123117 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:34.125970 (+ 2853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.1:59259: BlockingRecv error: recv got EOF from 127.2.32.1:59259 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
Jun 21 05:21:34 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:34 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:34.251003 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:34.243888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.2:34365)
0621 05:21:34.244031 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:34.244043 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:34.244108 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:34.244318 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:34.244319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:34.244476 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:34.244524 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:34.244526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:34.245216 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:34.245217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:34.245733 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:34.245735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:34.246174 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:34.246176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:34.246254 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:34.247577 (+ 1323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:34.247583 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:34.250926 (+ 3343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
W20260621 05:21:34.251313 11391 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:34.243983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:34365 (local address 127.2.32.62:40659)
0621 05:21:34.244061 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:34.244064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:34.244096 (+ 32us) server_negotiation.cc:407] Connection header received
0621 05:21:34.244173 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:34.244174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:34.244185 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:34.244230 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:34.244617 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:34.245122 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:34.245829 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:34.246032 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:34.246094 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:34.251237 (+ 5143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.2:34365: BlockingRecv error: recv got EOF from 127.2.32.2:34365 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 21 05:21:34 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:34 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:34.386126 11393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:34.380898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.3:54343)
0621 05:21:34.381131 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:34.381150 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:34.381244 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:34.381408 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:34.381409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:34.381599 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:34.381648 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:34.381649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:34.382336 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:34.382337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:34.382879 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:34.382881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:34.383350 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:34.383351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:34.383408 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:34.383936 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:34.383942 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:34.386069 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":230,"threads_started":1}
W20260621 05:21:34.386286 11391 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:34.380986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:54343 (local address 127.2.32.62:40659)
0621 05:21:34.381080 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:34.381082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:34.381221 (+ 139us) server_negotiation.cc:407] Connection header received
0621 05:21:34.381311 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:34.381312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:34.381326 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:34.381402 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:34.381728 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:34.382249 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:34.382983 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:34.383237 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:34.383351 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:34.386198 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.3:54343: BlockingRecv error: recv got EOF from 127.2.32.3:54343 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 21 05:21:35 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:35 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:35.132048 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:35.126326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.1:54099)
0621 05:21:35.126535 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:35.126560 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:35.126702 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:35.126878 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:35.126879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:35.127079 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:35.127230 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:35.127233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:35.127997 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:35.127998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:35.128585 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:35.128587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:35.129061 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:35.129062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:35.129103 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:35.129675 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:35.129680 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:35.131973 (+ 2293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":64,"threads_started":1}
W20260621 05:21:35.132243 11395 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:35.126535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:54099 (local address 127.2.32.62:40659)
0621 05:21:35.126703 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:35.126705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:35.126717 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:21:35.126766 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:35.126766 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:35.126779 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:35.126852 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:35.127357 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:35.127889 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:35.128727 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:35.128980 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:35.129054 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:35.132076 (+ 3022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.1:54099: BlockingRecv error: recv got EOF from 127.2.32.1:54099 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
Jun 21 05:21:35 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:35 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:35.257453 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:35.251818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.2:46733)
0621 05:21:35.252014 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:35.252036 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:35.252098 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:35.252311 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:35.252312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:35.252605 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:35.252674 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:35.252676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:35.253611 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:35.253612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:35.254387 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:35.254390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:35.254943 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:35.254943 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:35.255001 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:35.255594 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:35.255598 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:35.257349 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":128,"threads_started":1}
W20260621 05:21:35.257493 11395 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:35.252006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:46733 (local address 127.2.32.62:40659)
0621 05:21:35.252082 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:35.252084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:35.252094 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:21:35.252156 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:35.252156 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:35.252166 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:35.252235 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:35.252886 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:35.253462 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:35.254506 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:35.254851 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:35.254943 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:35.257430 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.2:46733: BlockingRecv error: recv got EOF from 127.2.32.2:46733 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 21 05:21:35 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:35 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:35.393636 11397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:35.386958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.3:34209)
0621 05:21:35.387212 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:35.387228 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:35.387294 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:35.387538 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:35.387540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:35.387770 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:35.387845 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:35.387848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:35.389237 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:35.389239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:35.389798 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:35.389800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:35.390254 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:35.390255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:35.390300 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:35.390880 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:35.390885 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:35.393577 (+ 2692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":81,"threads_started":1}
W20260621 05:21:35.393945 11395 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:35.387195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:34209 (local address 127.2.32.62:40659)
0621 05:21:35.387269 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:35.387271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:35.387295 (+ 24us) server_negotiation.cc:407] Connection header received
0621 05:21:35.387365 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:35.387366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:35.387380 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:35.387424 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:35.388012 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:35.389036 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:35.389936 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:35.390139 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:35.390228 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:35.393825 (+ 3597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.3:34209: BlockingRecv error: recv got EOF from 127.2.32.3:34209 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 21 05:21:36 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:36 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:36.138536 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:36.132559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.1:54305)
0621 05:21:36.132822 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:36.132841 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:36.132957 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:36.133209 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:36.133210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:36.133421 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:36.133495 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:36.133497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:36.134315 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:36.134316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:36.134968 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:36.134970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:36.135686 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:36.135688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:36.135780 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:36.136361 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:36.136366 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:36.138437 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":101,"threads_started":1}
W20260621 05:21:36.138686 11399 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:36.132657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:54305 (local address 127.2.32.62:40659)
0621 05:21:36.132768 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:36.132770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:36.132991 (+ 221us) server_negotiation.cc:407] Connection header received
0621 05:21:36.133045 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:36.133046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:36.133063 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:36.133192 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:36.133638 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:36.134196 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:36.135027 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:36.135560 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:36.135720 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:36.138586 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.1:54305: BlockingRecv error: recv got EOF from 127.2.32.1:54305 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":49,"threads_started":1}
Jun 21 05:21:36 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:36 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:36.263222 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:36.258313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.2:35563)
0621 05:21:36.258570 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:36.258590 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:36.258683 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:36.258855 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:36.258856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:36.259044 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:36.259090 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:36.259091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:36.259829 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:36.259830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:36.260344 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:36.260345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:36.260693 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:36.260694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:36.260746 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:36.261289 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:36.261293 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:36.263107 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":177,"threads_started":1}
W20260621 05:21:36.263275 11399 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:36.258468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:35563 (local address 127.2.32.62:40659)
0621 05:21:36.258549 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:36.258552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:36.258668 (+ 116us) server_negotiation.cc:407] Connection header received
0621 05:21:36.258759 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:36.258760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:36.258773 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:36.258847 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:36.259250 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:36.259747 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:36.260430 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:36.260620 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:36.260696 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:36.263211 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.2:35563: BlockingRecv error: recv got EOF from 127.2.32.2:35563 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 21 05:21:36 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:36 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:36.400842 11401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:36.394298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.3:41635)
0621 05:21:36.394500 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:36.394512 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:36.394584 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:36.394978 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:36.394979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:36.395220 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:36.395665 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:36.395671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:36.396504 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:36.396505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:36.397036 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:36.397038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:36.397452 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:36.397453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:36.397509 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:36.398122 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:36.398130 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:36.400781 (+ 2651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":130,"threads_started":1}
W20260621 05:21:36.401043 11399 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:36.394379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:41635 (local address 127.2.32.62:40659)
0621 05:21:36.394468 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:36.394470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:36.394594 (+ 124us) server_negotiation.cc:407] Connection header received
0621 05:21:36.394628 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:36.394628 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:36.394641 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:36.394817 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:36.395792 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:36.396360 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:36.397161 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:36.397341 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:36.397403 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:36.400980 (+ 3577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.3:41635: BlockingRecv error: recv got EOF from 127.2.32.3:41635 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 21 05:21:37 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:37 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:37.145135 11402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:37.139385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.1:34251)
0621 05:21:37.139631 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:37.139650 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:37.139748 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:37.139892 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:37.139894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:37.140087 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:37.140134 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:37.140135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:37.140942 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:37.140943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:37.141684 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:37.141686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:37.142156 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:37.142158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:37.142218 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:37.142688 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:37.142693 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:37.145061 (+ 2368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":119,"threads_started":1}
W20260621 05:21:37.145279 11403 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:37.139484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:34251 (local address 127.2.32.62:40659)
0621 05:21:37.139729 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:37.139731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:37.139744 (+ 13us) server_negotiation.cc:407] Connection header received
0621 05:21:37.139800 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:37.139801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:37.139814 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:37.139888 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:37.140248 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:37.140852 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:37.141774 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:37.142071 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:37.142153 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:37.145161 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.1:34251: BlockingRecv error: recv got EOF from 127.2.32.1:34251 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":148,"threads_started":1}
Jun 21 05:21:37 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:37 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:37.268477 11404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:37.263801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.2:37737)
0621 05:21:37.263956 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:37.263975 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:37.264021 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:37.264173 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:37.264174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:37.264333 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:37.264382 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:37.264383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:37.265085 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:37.265086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:37.265616 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:37.265617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:37.266024 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:37.266025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:37.266070 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:37.266547 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:37.266552 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:37.268418 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
W20260621 05:21:37.268908 11403 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:37.263916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:37737 (local address 127.2.32.62:40659)
0621 05:21:37.264010 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:37.264012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:37.264021 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:21:37.264073 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:37.264074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:37.264085 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:37.264144 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:37.264476 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:37.264992 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:37.265708 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:37.265931 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:37.265996 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:37.268851 (+ 2855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.2:37737: BlockingRecv error: recv got EOF from 127.2.32.2:37737 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 21 05:21:37 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:37 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:37.406819 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:37.401539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.3:52497)
0621 05:21:37.401737 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:37.401749 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:37.401843 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:37.402017 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:37.402018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:37.402238 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:37.402330 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:37.402332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:37.403294 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:37.403296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:37.403884 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:37.403886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:37.404340 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:37.404341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:37.404402 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:37.404901 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:37.404906 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:37.406754 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":109,"threads_started":1}
W20260621 05:21:37.407032 11403 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:37.401596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:52497 (local address 127.2.32.62:40659)
0621 05:21:37.401705 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:37.401709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:37.401828 (+ 119us) server_negotiation.cc:407] Connection header received
0621 05:21:37.401908 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:37.401910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:37.401930 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:37.402011 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:37.402416 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:37.403092 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:37.403995 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:37.404246 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:37.404326 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:37.406855 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.3:52497: BlockingRecv error: recv got EOF from 127.2.32.3:52497 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 21 05:21:38 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:38 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:38.156968 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:38.145750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.1:38423)
0621 05:21:38.145874 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:38.145900 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:38.145956 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:38.146328 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:38.146330 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:38.146568 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:38.146725 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:38.146729 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:38.147673 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:38.147674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:38.148616 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:38.148620 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:38.149110 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:38.149112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:38.149178 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:38.154376 (+ 5198us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:38.154383 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:38.156898 (+ 2515us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
W20260621 05:21:38.157157 11407 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:38.145911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:38423 (local address 127.2.32.62:40659)
0621 05:21:38.146084 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:38.146086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:38.146096 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:21:38.146122 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:38.146122 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:38.146134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:38.146220 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:38.146869 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:38.147542 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:38.148757 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:38.148981 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:38.149054 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:38.157088 (+ 8034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.1:38423: BlockingRecv error: recv got EOF from 127.2.32.1:38423 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":109,"threads_started":1}
Jun 21 05:21:38 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:38 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:38.275444 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:38.269246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.2:47471)
0621 05:21:38.269421 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:38.269440 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:38.269549 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:38.269828 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:38.269829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:38.270126 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:38.270188 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:38.270190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:38.271233 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:38.271234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:38.271855 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:38.271857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:38.272270 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:38.272272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:38.272322 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:38.272959 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:38.272969 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:38.275376 (+ 2407us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":102,"threads_started":1}
W20260621 05:21:38.275678 11407 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:38.269369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:47471 (local address 127.2.32.62:40659)
0621 05:21:38.269457 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:38.269460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:38.269567 (+ 107us) server_negotiation.cc:407] Connection header received
0621 05:21:38.269672 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:38.269673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:38.269687 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:38.269784 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:38.270266 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:38.271088 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:38.271957 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:38.272176 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:38.272245 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:38.275589 (+ 3344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.2:47471: BlockingRecv error: recv got EOF from 127.2.32.2:47471 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 21 05:21:38 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:38 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:38.413244 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:38.407533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.3:53819)
0621 05:21:38.407795 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:38.407809 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:38.407898 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:38.408154 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:38.408156 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:38.408355 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:38.408408 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:38.408409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:38.409241 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:38.409242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:38.410003 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:38.410006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:38.410425 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:38.410426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:38.410473 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:38.411061 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:38.411067 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:38.413171 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":201,"threads_started":1}
W20260621 05:21:38.413447 11407 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:38.407676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:53819 (local address 127.2.32.62:40659)
0621 05:21:38.407778 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:38.407780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:38.407940 (+ 160us) server_negotiation.cc:407] Connection header received
0621 05:21:38.407975 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:38.407976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:38.407993 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:38.408073 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:38.408538 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:38.409144 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:38.410117 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:38.410333 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:38.410406 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:38.413354 (+ 2948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.3:53819: BlockingRecv error: recv got EOF from 127.2.32.3:53819 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 21 05:21:39 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:39 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:39.163050 11410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:39.157541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.1:35389)
0621 05:21:39.157749 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:39.157761 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:39.157828 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:39.157973 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:39.157974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:39.158152 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:39.158201 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:39.158203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:39.158926 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:39.158927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:39.159561 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:39.159562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:39.159968 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:39.159970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:39.160017 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:39.160533 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:39.160540 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:39.162978 (+ 2438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":101,"threads_started":1}
W20260621 05:21:39.163247 11411 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:39.157653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:35389 (local address 127.2.32.62:40659)
0621 05:21:39.157760 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:39.157762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:39.157810 (+ 48us) server_negotiation.cc:407] Connection header received
0621 05:21:39.157875 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:39.157876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:39.157886 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:39.157953 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:39.158324 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:39.158827 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:39.159666 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:39.159861 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:39.159922 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:39.163126 (+ 3204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.1:35389: BlockingRecv error: recv got EOF from 127.2.32.1:35389 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":50,"threads_started":1}
Jun 21 05:21:39 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:39 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:39.281316 11412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:39.276216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.2:42193)
0621 05:21:39.276369 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:39.276383 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:39.276484 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:39.276872 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:39.276873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:39.277107 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:39.277175 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:39.277183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:39.277920 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:39.277921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:39.278468 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:39.278469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:39.278889 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:39.278890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:39.278938 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:39.279389 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:39.279393 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:39.281260 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
W20260621 05:21:39.281489 11411 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:39.276295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:42193 (local address 127.2.32.62:40659)
0621 05:21:39.276364 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:39.276366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:39.276563 (+ 197us) server_negotiation.cc:407] Connection header received
0621 05:21:39.276733 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:39.276734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:39.276755 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:39.276835 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:39.277297 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:39.277834 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:39.278573 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:39.278769 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:39.278835 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:39.281392 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.2:42193: BlockingRecv error: recv got EOF from 127.2.32.2:42193 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 21 05:21:39 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:39 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:39.419518 11413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:39.413906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.3:55515)
0621 05:21:39.414144 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:39.414165 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:39.414240 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:39.414469 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:39.414471 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:39.414726 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:39.414862 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:39.414870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:39.415661 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:39.415662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:39.416468 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:39.416471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:39.416852 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:39.416853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:39.416899 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:39.417409 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:39.417413 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:39.419459 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":143,"threads_started":1}
W20260621 05:21:39.419667 11411 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:39.414049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:55515 (local address 127.2.32.62:40659)
0621 05:21:39.414128 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:39.414130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:39.414253 (+ 123us) server_negotiation.cc:407] Connection header received
0621 05:21:39.414303 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:39.414304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:39.414320 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:39.414393 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:39.414993 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:39.415540 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:39.416565 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:39.416746 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:39.416810 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:39.419607 (+ 2797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.3:55515: BlockingRecv error: recv got EOF from 127.2.32.3:55515 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 21 05:21:40 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:40 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:40.169410 11414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:40.163577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.1:45181)
0621 05:21:40.163813 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:40.163825 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:40.163895 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:40.164017 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:40.164018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:40.164195 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:40.164244 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:40.164246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:40.164963 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:40.164964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:40.165537 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:40.165538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:40.165911 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:40.165912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:40.165960 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:40.166703 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:40.166736 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:40.169350 (+ 2614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":113,"threads_started":1}
W20260621 05:21:40.169677 11415 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:40.163701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:45181 (local address 127.2.32.62:40659)
0621 05:21:40.163827 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:40.163828 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:40.163900 (+ 72us) server_negotiation.cc:407] Connection header received
0621 05:21:40.163943 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:40.163943 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:40.163955 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:40.164013 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:40.164373 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:40.164874 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:40.165623 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:40.165826 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:40.165883 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:40.169565 (+ 3682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.1:45181: BlockingRecv error: recv got EOF from 127.2.32.1:45181 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
Jun 21 05:21:40 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:40 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:40.287066 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:40.281895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.2:58305)
0621 05:21:40.282080 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:40.282100 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:40.282160 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:40.282286 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:40.282287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:40.282454 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:40.282503 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:40.282504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:40.283297 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:40.283299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:40.283995 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:40.283997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:40.284509 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:40.284510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:40.284561 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:40.285098 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:40.285103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:40.286987 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":60,"threads_started":1}
W20260621 05:21:40.287186 11415 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:40.282028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:58305 (local address 127.2.32.62:40659)
0621 05:21:40.282096 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:40.282098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:40.282143 (+ 45us) server_negotiation.cc:407] Connection header received
0621 05:21:40.282204 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:40.282205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:40.282216 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:40.282264 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:40.282569 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:40.283089 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:40.284135 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:40.284364 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:40.284480 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:40.287096 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.2:58305: BlockingRecv error: recv got EOF from 127.2.32.2:58305 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 21 05:21:40 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:40 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:40.425385 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:40.420152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.3:51335)
0621 05:21:40.420329 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:40.420343 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:40.420427 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:40.420585 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:40.420586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:40.420763 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:40.420816 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:40.420818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:40.421475 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:40.421476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:40.422027 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:40.422029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:40.422375 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:40.422376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:40.422432 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:40.422952 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:40.422957 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:40.425285 (+ 2328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":81,"threads_started":1}
W20260621 05:21:40.425422 11415 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:40.420379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:51335 (local address 127.2.32.62:40659)
0621 05:21:40.420453 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:40.420455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:40.420465 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:21:40.420489 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:40.420490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:40.420511 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:40.420576 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:40.420898 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:40.421393 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:40.422113 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:40.422302 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:40.422377 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:40.425361 (+ 2984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.3:51335: BlockingRecv error: recv got EOF from 127.2.32.3:51335 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 21 05:21:41 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:41 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:41.176333 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:41.169889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.1:44793)
0621 05:21:41.170021 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:41.170036 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:41.170114 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:41.170383 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:41.170384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:41.170613 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:41.170673 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:41.170676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:41.171773 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:41.171774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:41.172306 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:41.172307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:41.172934 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:41.172936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:41.173015 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:41.173691 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:41.173696 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:41.176214 (+ 2518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
W20260621 05:21:41.176473 11419 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:41.170060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:44793 (local address 127.2.32.62:40659)
0621 05:21:41.170192 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:41.170194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:41.170202 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:21:41.170223 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:41.170224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:41.170239 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:41.170304 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:41.170761 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:41.171684 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:41.172393 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:41.172784 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:41.172874 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:41.176350 (+ 3476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.1:44793: BlockingRecv error: recv got EOF from 127.2.32.1:44793 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":41,"threads_started":1}
Jun 21 05:21:41 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:41 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:41.292984 11420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:41.287792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.2:54667)
0621 05:21:41.287994 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:41.288013 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:41.288111 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:41.288368 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:41.288369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:41.288613 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:41.288686 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:41.288688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:41.289514 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:41.289515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:41.290097 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:41.290100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:41.290521 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:41.290521 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:41.290565 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:41.291168 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:41.291173 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:41.292926 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":132,"threads_started":1}
W20260621 05:21:41.293228 11419 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:41.287926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:54667 (local address 127.2.32.62:40659)
0621 05:21:41.287991 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:41.287993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:41.288158 (+ 165us) server_negotiation.cc:407] Connection header received
0621 05:21:41.288203 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:41.288204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:41.288221 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:41.288318 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:41.288812 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:41.289389 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:41.290212 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:41.290434 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:41.290502 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:41.293143 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.2:54667: BlockingRecv error: recv got EOF from 127.2.32.2:54667 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 21 05:21:41 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:41 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:41.431377 11421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:41.425988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.3:42283)
0621 05:21:41.426166 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:41.426186 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:41.426238 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:41.426492 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:41.426494 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:41.426781 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:41.426831 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:41.426834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:41.427755 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:41.427756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:41.428301 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:41.428302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:41.429000 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:41.429001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:41.429068 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:41.429617 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:41.429622 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:41.431319 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":115,"threads_started":1}
W20260621 05:21:41.431699 11419 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:41.426168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:42283 (local address 127.2.32.62:40659)
0621 05:21:41.426222 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:41.426224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:41.426233 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:21:41.426307 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:41.426308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:41.426321 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:41.426383 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:41.426979 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:41.427627 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:41.428446 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:41.428818 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:41.428924 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:41.431625 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.3:42283: BlockingRecv error: recv got EOF from 127.2.32.3:42283 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 21 05:21:42 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:42 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:42.182729 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:42.176992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.1:34069)
0621 05:21:42.177262 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:42.177274 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:42.177350 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:42.177548 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:42.177549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:42.177729 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:42.177789 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:42.177791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:42.178517 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:42.178518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:42.179047 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:42.179048 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:42.179476 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:42.179477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:42.179524 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:42.180072 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:42.180076 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:42.182670 (+ 2594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":195,"threads_started":1}
W20260621 05:21:42.183171 11422 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:42.177071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:34069 (local address 127.2.32.62:40659)
0621 05:21:42.177232 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:42.177235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:42.177378 (+ 143us) server_negotiation.cc:407] Connection header received
0621 05:21:42.177403 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:42.177403 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:42.177415 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:42.177490 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:42.177918 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:42.178427 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:42.179160 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:42.179346 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:42.179412 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:42.182923 (+ 3511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.1:34069: BlockingRecv error: recv got EOF from 127.2.32.1:34069 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
Jun 21 05:21:42 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:42 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:42.299307 11424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:42.293658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.2:59645)
0621 05:21:42.293906 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:42.293924 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:42.294059 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:42.294219 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:42.294219 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:42.294405 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:42.294454 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:42.294456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:42.295176 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:42.295177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:42.295711 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:42.295713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:42.296307 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:42.296309 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:42.296386 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:42.297049 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:42.297055 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:42.299218 (+ 2163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":133,"threads_started":1}
W20260621 05:21:42.299400 11422 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:42.293826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:59645 (local address 127.2.32.62:40659)
0621 05:21:42.293932 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:42.293935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:42.294035 (+ 100us) server_negotiation.cc:407] Connection header received
0621 05:21:42.294118 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:42.294119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:42.294133 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:42.294213 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:42.294535 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:42.295034 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:42.295798 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:42.296140 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:42.296289 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:42.299336 (+ 3047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.2:59645: BlockingRecv error: recv got EOF from 127.2.32.2:59645 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 21 05:21:42 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:42 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:42.437163 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:42.431859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.3:50455)
0621 05:21:42.432026 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:42.432037 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:42.432104 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:42.432350 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:42.432352 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:42.432564 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:42.432639 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:42.432642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:42.433579 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:42.433580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:42.434158 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:42.434160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:42.434557 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:42.434558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:42.434636 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:42.435245 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:42.435250 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:42.437105 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":77,"threads_started":1}
W20260621 05:21:42.437363 11422 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:42.431960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:50455 (local address 127.2.32.62:40659)
0621 05:21:42.432011 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:42.432013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:42.432115 (+ 102us) server_negotiation.cc:407] Connection header received
0621 05:21:42.432148 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:42.432149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:42.432160 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:42.432207 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:42.432840 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:42.433457 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:42.434263 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:42.434452 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:42.434520 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:42.437281 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.3:50455: BlockingRecv error: recv got EOF from 127.2.32.3:50455 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 21 05:21:43 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:43 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:43.193414 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:43.183480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.1:45657)
0621 05:21:43.183670 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:43.183682 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:43.183742 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:43.183949 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:43.183950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:43.184113 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:43.184161 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:43.184162 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:43.184888 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:43.184889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:43.185710 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:43.185712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:43.186063 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:43.186064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:43.186101 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:43.191500 (+ 5399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:43.191505 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:43.193322 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":89,"threads_started":1}
W20260621 05:21:43.193727 11427 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:43.183576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:45657 (local address 127.2.32.62:40659)
0621 05:21:43.183715 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:43.183716 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:43.183728 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:21:43.183851 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:43.183852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:43.183865 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:43.183931 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:43.184280 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:43.184783 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:43.185794 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:43.185996 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:43.186057 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:43.193638 (+ 7581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.1:45657: BlockingRecv error: recv got EOF from 127.2.32.1:45657 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
Jun 21 05:21:43 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:43 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:43.304858 11428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:43.299906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.2:57585)
0621 05:21:43.300089 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:43.300101 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:43.300161 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:43.300317 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:43.300318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:43.300478 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:43.300526 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:43.300527 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:43.301227 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:43.301228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:43.301759 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:43.301761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:43.302306 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:43.302307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:43.302350 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:43.302867 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:43.302871 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:43.304771 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":84,"threads_started":1}
W20260621 05:21:43.305178 11427 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:43.300063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:57585 (local address 127.2.32.62:40659)
0621 05:21:43.300152 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:43.300154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:43.300162 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:21:43.300229 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:43.300230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:43.300240 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:43.300291 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:43.300615 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:43.301147 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:43.301913 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:43.302176 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:43.302277 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:43.305000 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.2:57585: BlockingRecv error: recv got EOF from 127.2.32.2:57585 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 21 05:21:43 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:43 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:43.442663 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:43.437781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.3:38081)
0621 05:21:43.437927 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:43.437944 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:43.438004 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:43.438143 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:43.438143 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:43.438318 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:43.438374 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:43.438375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:43.439007 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:43.439008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:43.439555 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:43.439557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:43.439919 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:43.439920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:43.439971 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:43.440446 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:43.440457 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:43.442559 (+ 2102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":57,"threads_started":1}
W20260621 05:21:43.442685 11427 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:43.437874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:38081 (local address 127.2.32.62:40659)
0621 05:21:43.437948 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:43.437950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:43.438002 (+ 52us) server_negotiation.cc:407] Connection header received
0621 05:21:43.438064 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:43.438065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:43.438077 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:43.438123 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:43.438441 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:43.438930 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:43.439654 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:43.439835 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:43.439891 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:43.442623 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.3:38081: BlockingRecv error: recv got EOF from 127.2.32.3:38081 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 21 05:21:44 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:44 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:44.199489 11430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:44.194036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.1:48717)
0621 05:21:44.194339 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:44.194361 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:44.194440 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:44.194598 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:44.194599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:44.194781 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:44.194835 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:44.194836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:44.195795 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:44.195797 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:44.196387 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:44.196389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:44.196903 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:44.196904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:44.196967 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:44.197546 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:44.197551 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:44.199434 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":161,"threads_started":1}
W20260621 05:21:44.199638 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:44.194252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:48717 (local address 127.2.32.62:40659)
0621 05:21:44.194417 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:44.194419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:44.194429 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:21:44.194502 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:44.194503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:44.194513 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:44.194576 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:44.194955 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:44.195684 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:44.196532 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:44.196794 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:44.196867 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:44.199539 (+ 2672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.1:48717: BlockingRecv error: recv got EOF from 127.2.32.1:48717 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":96,"threads_started":1}
Jun 21 05:21:44 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:44 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:44.310859 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:44.305525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.2:43801)
0621 05:21:44.305699 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:44.305710 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:44.305770 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:44.306081 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:44.306082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:44.306254 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:44.306310 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:44.306312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:44.307048 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:44.307049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:44.307617 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:44.307619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:44.308064 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:44.308065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:44.308106 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:44.308637 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:44.308643 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:44.310751 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":113,"threads_started":1}
W20260621 05:21:44.310930 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:44.305619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:43801 (local address 127.2.32.62:40659)
0621 05:21:44.305689 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:44.305691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:44.305874 (+ 183us) server_negotiation.cc:407] Connection header received
0621 05:21:44.305923 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:44.305925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:44.305943 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:44.306043 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:44.306425 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:44.306957 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:44.307731 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:44.307963 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:44.308053 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:44.310877 (+ 2824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.2:43801: BlockingRecv error: recv got EOF from 127.2.32.2:43801 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 21 05:21:44 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:44 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.3@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:44.448310 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:44.443337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.3:39135)
0621 05:21:44.443459 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:44.443469 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:44.443517 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:44.443667 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:44.443668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:44.443819 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:44.443867 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:44.443869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:44.444561 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:44.444562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:44.445157 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:44.445159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:44.445543 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:44.445544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:44.445590 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:44.446103 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:44.446108 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:44.448204 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":65,"threads_started":1}
W20260621 05:21:44.448334 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:44.443422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:39135 (local address 127.2.32.62:40659)
0621 05:21:44.443495 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:44.443497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:44.443509 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:21:44.443580 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:44.443580 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:44.443592 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:44.443647 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:44.443963 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:44.444463 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:44.445273 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:44.445451 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:44.445514 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:44.448274 (+ 2760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.3:39135: BlockingRecv error: recv got EOF from 127.2.32.3:39135 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 21 05:21:45 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:45 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.1@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:45.205626 11434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:45.200140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.1:45227)
0621 05:21:45.200381 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:45.200403 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:45.200501 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:45.200695 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:45.200696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:45.200875 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:45.200936 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:45.200938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.201617 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.201618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:45.202197 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:45.202199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.202761 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.202763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:45.202805 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:45.203424 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:45.203429 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:45.205527 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":140,"threads_started":1}
W20260621 05:21:45.205725 11435 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:45.200300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:45227 (local address 127.2.32.62:40659)
0621 05:21:45.200423 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:45.200425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:45.200473 (+ 48us) server_negotiation.cc:407] Connection header received
0621 05:21:45.200566 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:45.200567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:45.200577 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:45.200636 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:45.201017 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.201517 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.202323 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.202672 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.202743 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:45.205642 (+ 2899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.1:45227: BlockingRecv error: recv got EOF from 127.2.32.1:45227 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
Jun 21 05:21:45 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 05:21:45 dist-test-slave-b357 krb5kdc[10376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.32.2@KRBTEST.COM for oryx/127.2.32.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 05:21:45.317921 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:45.311701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40659 (local address 127.2.32.2:54873)
0621 05:21:45.311928 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:45.311943 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:45.312046 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:45.312240 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:45.312241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:45.312435 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:45.312495 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:45.312497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.313294 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.313295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:45.313941 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:45.313944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.314409 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.314410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:45.314518 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:45.315252 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:45.315258 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:45.317842 (+ 2584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.32.62:40659: Server oryx/127.2.32.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":155,"threads_started":1}
W20260621 05:21:45.318120 11435 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:45.311857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:54873 (local address 127.2.32.62:40659)
0621 05:21:45.311953 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:45.311955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:45.312062 (+ 107us) server_negotiation.cc:407] Connection header received
0621 05:21:45.312104 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:45.312105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:45.312125 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:45.312189 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:45.312626 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.313201 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.314047 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.314246 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.314314 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:45.318041 (+ 3727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.32.2:54873: BlockingRecv error: recv got EOF from 127.2.32.2:54873 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 05:21:45.350008 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 10959
I20260621 05:21:45.355906 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 11111
I20260621 05:21:45.361075 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 11245
I20260621 05:21:45.365276 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 10889
2026-06-21T05:21:45Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17524 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-21T05:21:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:21:45Z Disabled control of system clock
I20260621 05:21:45.393484 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:42067
--webserver_interface=127.2.32.62
--webserver_port=0
--builtin_ntp_servers=127.2.32.20:43509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:42067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260621 05:21:45.474603 11440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:45.474890 11440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:45.474915 11440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:45.476457 11440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:21:45.476516 11440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:45.476528 11440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:45.476542 11440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:21:45.476555 11440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:21:45.478484 11440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:43509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:42067
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:42067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:45.478729 11440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:45.478962 11440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:45.483996 11446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:45.484025 11440 server_base.cc:1034] running on GCE node
W20260621 05:21:45.484069 11445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:45.484000 11448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:45.484652 11440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:45.484951 11440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:45.486158 11440 hybrid_clock.cc:648] HybridClock initialized: now 1782019305486138 us; error 45 us; skew 500 ppm
I20260621 05:21:45.487537 11440 webserver.cc:460] Webserver started at http://127.2.32.62:40991/ using document root <none> and password file <none>
I20260621 05:21:45.487772 11440 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:45.487840 11440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:45.488000 11440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:21:45.489076 11440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "f07ea7f6b71b440eb645a1dbc6735ae6"
format_stamp: "Formatted at 2026-06-21 05:21:45 on dist-test-slave-b357"
server_key: "ca628866265c05d8b3008a9c41de5584"
server_key_iv: "243e1064db28880efdcbc7e49b198311"
server_key_version: "encryptionkey@0"
I20260621 05:21:45.489431 11440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "f07ea7f6b71b440eb645a1dbc6735ae6"
format_stamp: "Formatted at 2026-06-21 05:21:45 on dist-test-slave-b357"
server_key: "ca628866265c05d8b3008a9c41de5584"
server_key_iv: "243e1064db28880efdcbc7e49b198311"
server_key_version: "encryptionkey@0"
I20260621 05:21:45.490741 11440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:21:45.491891 11454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:45.492233 11440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:21:45.492380 11440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "f07ea7f6b71b440eb645a1dbc6735ae6"
format_stamp: "Formatted at 2026-06-21 05:21:45 on dist-test-slave-b357"
server_key: "ca628866265c05d8b3008a9c41de5584"
server_key_iv: "243e1064db28880efdcbc7e49b198311"
server_key_version: "encryptionkey@0"
I20260621 05:21:45.492481 11440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:45.500921 11440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:45.501216 11440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:45.501372 11440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:45.505602 11440 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:42067
I20260621 05:21:45.505640 11506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:42067 every 8 connection(s)
I20260621 05:21:45.505981 11440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:21:45.506724 11507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:45.508631 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 11440
I20260621 05:21:45.509007 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:21:45.509197 2176 external_mini_cluster.cc:1428] Setting key e048a24c0c762ff2992aa0b66bf47fae
I20260621 05:21:45.509414 11507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6: Bootstrap starting.
I20260621 05:21:45.510272 11507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:45.510604 11507 log.cc:824] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6: Log is configured to *not* fsync() on all Append() calls
I20260621 05:21:45.511380 11507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6: No bootstrap required, opened a new log
I20260621 05:21:45.512991 11507 raft_consensus.cc:348] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07ea7f6b71b440eb645a1dbc6735ae6" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 42067 } }
I20260621 05:21:45.513166 11507 raft_consensus.cc:374] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:45.513200 11507 raft_consensus.cc:729] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f07ea7f6b71b440eb645a1dbc6735ae6, State: Initialized, Role: FOLLOWER
I20260621 05:21:45.513340 11507 consensus_queue.cc:260] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07ea7f6b71b440eb645a1dbc6735ae6" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 42067 } }
I20260621 05:21:45.513433 11507 raft_consensus.cc:388] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:21:45.513461 11507 raft_consensus.cc:482] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:21:45.513499 11507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:45.514351 11507 raft_consensus.cc:504] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07ea7f6b71b440eb645a1dbc6735ae6" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 42067 } }
I20260621 05:21:45.514479 11507 leader_election.cc:302] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f07ea7f6b71b440eb645a1dbc6735ae6; no voters:
I20260621 05:21:45.514696 11507 leader_election.cc:288] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:21:45.514747 11512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:21:45.514900 11512 raft_consensus.cc:686] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6 [term 1 LEADER]: Becoming Leader. State: Replica: f07ea7f6b71b440eb645a1dbc6735ae6, State: Running, Role: LEADER
I20260621 05:21:45.515002 11507 sys_catalog.cc:564] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:21:45.515064 11512 consensus_queue.cc:237] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07ea7f6b71b440eb645a1dbc6735ae6" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 42067 } }
I20260621 05:21:45.516074 11510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:45.510068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42464 (local address 127.2.32.62:42067)
0621 05:21:45.510387 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:45.510405 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:45.510438 (+ 33us) server_negotiation.cc:407] Connection header received
0621 05:21:45.511082 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:45.511091 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:45.511228 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:45.511372 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:45.511671 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.512659 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.513813 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.514354 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.514505 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:45.515286 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:45.515291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:45.515292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:21:45.515486 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:45.515520 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:21:45.515531 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:45.515609 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:45.515789 (+ 180us) server_negotiation.cc:299] Negotiation successful
0621 05:21:45.515875 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":172,"threads_started":1}
I20260621 05:21:45.516171 11514 sys_catalog.cc:455] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f07ea7f6b71b440eb645a1dbc6735ae6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07ea7f6b71b440eb645a1dbc6735ae6" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 42067 } } }
I20260621 05:21:45.516242 11514 sys_catalog.cc:458] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6 [sys.catalog]: This master's current role is: LEADER
I20260621 05:21:45.516656 11513 sys_catalog.cc:455] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f07ea7f6b71b440eb645a1dbc6735ae6. Latest consensus state: current_term: 1 leader_uuid: "f07ea7f6b71b440eb645a1dbc6735ae6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07ea7f6b71b440eb645a1dbc6735ae6" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 42067 } } }
I20260621 05:21:45.516714 11513 sys_catalog.cc:458] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6 [sys.catalog]: This master's current role is: LEADER
W20260621 05:21:45.517014 11527 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:21:45.517055 11527 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:21:45.517118 11526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:21:45.517275 11526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:21:45.518606 11526 catalog_manager.cc:1347] Generated new cluster ID: 19861ece8a36477288d52afb0ff6968f
I20260621 05:21:45.518657 11526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:21:45.540611 11526 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:21:45.541178 11526 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:21:45.549388 11526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f07ea7f6b71b440eb645a1dbc6735ae6: Generated new TSK 0
I20260621 05:21:45.549717 11526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:21:45.557292 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:42067
--builtin_ntp_servers=127.2.32.20:43509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260621 05:21:45.646637 11531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:45.646834 11531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:45.646855 11531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:45.648033 11531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:45.648075 11531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:45.648099 11531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:21:45.649434 11531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:43509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:42067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:45.649669 11531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:45.649950 11531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:45.655436 11537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:45.655445 11539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:45.655447 11536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:45.655702 11531 server_base.cc:1034] running on GCE node
I20260621 05:21:45.655879 11531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:45.656054 11531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:45.657243 11531 hybrid_clock.cc:648] HybridClock initialized: now 1782019305657229 us; error 26 us; skew 500 ppm
I20260621 05:21:45.658490 11531 webserver.cc:460] Webserver started at http://127.2.32.1:39803/ using document root <none> and password file <none>
I20260621 05:21:45.658727 11531 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:45.658777 11531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:45.658946 11531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:21:45.660030 11531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "de251f07620542d4b72561b8b9215eb6"
format_stamp: "Formatted at 2026-06-21 05:21:45 on dist-test-slave-b357"
server_key: "028ed9df8260c073c85f99faa7a982b1"
server_key_iv: "74724523f2a5e2a249ad451647132478"
server_key_version: "encryptionkey@0"
I20260621 05:21:45.660383 11531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "de251f07620542d4b72561b8b9215eb6"
format_stamp: "Formatted at 2026-06-21 05:21:45 on dist-test-slave-b357"
server_key: "028ed9df8260c073c85f99faa7a982b1"
server_key_iv: "74724523f2a5e2a249ad451647132478"
server_key_version: "encryptionkey@0"
I20260621 05:21:45.661679 11531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:21:45.662519 11545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:45.662693 11531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:45.662781 11531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "de251f07620542d4b72561b8b9215eb6"
format_stamp: "Formatted at 2026-06-21 05:21:45 on dist-test-slave-b357"
server_key: "028ed9df8260c073c85f99faa7a982b1"
server_key_iv: "74724523f2a5e2a249ad451647132478"
server_key_version: "encryptionkey@0"
I20260621 05:21:45.662833 11531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:45.675984 11531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:45.676227 11531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:45.676338 11531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:45.676514 11531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:21:45.676908 11531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:21:45.676944 11531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:45.676966 11531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:21:45.676980 11531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:45.682986 11531 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:41075
I20260621 05:21:45.683076 11658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:41075 every 8 connection(s)
I20260621 05:21:45.683403 11531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:21:45.689280 11510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:45.684233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:44643 (local address 127.2.32.62:42067)
0621 05:21:45.684301 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:45.684303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:45.684536 (+ 233us) server_negotiation.cc:407] Connection header received
0621 05:21:45.684998 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:45.684999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:45.685009 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:45.685072 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:45.685769 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.686271 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.687443 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.687624 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.687702 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:45.688943 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:45.688945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:45.688945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:21:45.688949 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:45.688966 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:21:45.688970 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:45.689041 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:45.689199 (+ 158us) server_negotiation.cc:299] Negotiation successful
0621 05:21:45.689220 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 05:21:45.689514 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:45.684159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:42067 (local address 127.2.32.1:44643)
0621 05:21:45.684395 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:45.684433 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:45.684912 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:45.685134 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:45.685137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:45.685144 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:45.685678 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:45.685682 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.686379 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.686380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:45.687322 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:45.687325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.687728 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.687729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:45.687862 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:45.688713 (+ 851us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:21:45.688714 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:45.688762 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:21:45.688801 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:21:45.689083 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:21:45.689090 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:21:45.689092 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:21:45.689175 (+ 83us) client_negotiation.cc:240] Negotiation successful
0621 05:21:45.689265 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":95,"threads_started":1}
I20260621 05:21:45.690110 11659 heartbeater.cc:347] Connected to a master server at 127.2.32.62:42067
I20260621 05:21:45.690230 11659 heartbeater.cc:464] Registering TS with master...
I20260621 05:21:45.690424 11659 heartbeater.cc:511] Master 127.2.32.62:42067 requested a full tablet report, sending...
I20260621 05:21:45.691349 11470 ts_manager.cc:194] Registered new tserver with Master: de251f07620542d4b72561b8b9215eb6 (127.2.32.1:41075)
I20260621 05:21:45.692198 11470 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.32.1:44643
I20260621 05:21:45.693216 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 11531
I20260621 05:21:45.693307 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:21:45.693616 2176 external_mini_cluster.cc:1428] Setting key 28a4f3f5a84aea59e275b3d08d83a89b
I20260621 05:21:45.694828 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:42067
--builtin_ntp_servers=127.2.32.20:43509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260621 05:21:45.774726 11662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:45.774923 11662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:45.774945 11662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:45.776108 11662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:45.776154 11662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:45.776178 11662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:21:45.777483 11662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:43509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:42067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:45.777719 11662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:45.778021 11662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:45.783080 11670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:45.783032 11668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:45.783424 11662 server_base.cc:1034] running on GCE node
W20260621 05:21:45.783448 11667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:45.783661 11662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:45.783847 11662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:45.785017 11662 hybrid_clock.cc:648] HybridClock initialized: now 1782019305785008 us; error 32 us; skew 500 ppm
I20260621 05:21:45.786185 11662 webserver.cc:460] Webserver started at http://127.2.32.2:41489/ using document root <none> and password file <none>
I20260621 05:21:45.786429 11662 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:45.786500 11662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:45.786651 11662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:21:45.787724 11662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "03ec6a0fe62646099d8510e7dd6c63c0"
format_stamp: "Formatted at 2026-06-21 05:21:45 on dist-test-slave-b357"
server_key: "54d9120c252c0c95a7b3b60c8de00cc4"
server_key_iv: "6de9a1e17eaad22c9b9df8aa767a28ae"
server_key_version: "encryptionkey@0"
I20260621 05:21:45.788093 11662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "03ec6a0fe62646099d8510e7dd6c63c0"
format_stamp: "Formatted at 2026-06-21 05:21:45 on dist-test-slave-b357"
server_key: "54d9120c252c0c95a7b3b60c8de00cc4"
server_key_iv: "6de9a1e17eaad22c9b9df8aa767a28ae"
server_key_version: "encryptionkey@0"
I20260621 05:21:45.789727 11662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260621 05:21:45.790848 11676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:45.791193 11662 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:45.791311 11662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "03ec6a0fe62646099d8510e7dd6c63c0"
format_stamp: "Formatted at 2026-06-21 05:21:45 on dist-test-slave-b357"
server_key: "54d9120c252c0c95a7b3b60c8de00cc4"
server_key_iv: "6de9a1e17eaad22c9b9df8aa767a28ae"
server_key_version: "encryptionkey@0"
I20260621 05:21:45.791465 11662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:45.805117 11662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:45.805521 11662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:45.805699 11662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:45.805976 11662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:21:45.806372 11662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:21:45.806435 11662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:45.806483 11662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:21:45.806514 11662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:45.814150 11662 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:44869
I20260621 05:21:45.814234 11789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:44869 every 8 connection(s)
I20260621 05:21:45.814632 11662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:21:45.820113 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 11662
I20260621 05:21:45.820211 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:21:45.820930 2176 external_mini_cluster.cc:1428] Setting key 7ef338260f0626bf8d999c26a7ca26ee
I20260621 05:21:45.821324 11510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:45.815689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:36193 (local address 127.2.32.62:42067)
0621 05:21:45.815792 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:45.815795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:45.816044 (+ 249us) server_negotiation.cc:407] Connection header received
0621 05:21:45.816838 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:45.816840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:45.816854 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:45.816918 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:45.817835 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.818508 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.819650 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.819852 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.819939 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:45.821041 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:45.821042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:45.821042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:21:45.821047 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:45.821060 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:21:45.821064 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:45.821119 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:45.821222 (+ 103us) server_negotiation.cc:299] Negotiation successful
0621 05:21:45.821246 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 05:21:45.821532 11792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:45.815575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:42067 (local address 127.2.32.2:36193)
0621 05:21:45.815877 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:45.815920 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:45.816709 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:45.817002 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:45.817007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:45.817029 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:45.817656 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:45.817667 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.818631 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.818632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:45.819547 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:45.819549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.819948 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.819950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:45.820077 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:45.820855 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:21:45.820856 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:45.820890 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:21:45.820914 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:21:45.821126 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:21:45.821130 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:21:45.821133 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:21:45.821222 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 05:21:45.821324 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":143,"threads_started":1}
I20260621 05:21:45.822079 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:42067
--builtin_ntp_servers=127.2.32.20:43509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260621 05:21:45.823365 11790 heartbeater.cc:347] Connected to a master server at 127.2.32.62:42067
I20260621 05:21:45.823482 11790 heartbeater.cc:464] Registering TS with master...
I20260621 05:21:45.823688 11790 heartbeater.cc:511] Master 127.2.32.62:42067 requested a full tablet report, sending...
I20260621 05:21:45.824206 11470 ts_manager.cc:194] Registered new tserver with Master: 03ec6a0fe62646099d8510e7dd6c63c0 (127.2.32.2:44869)
I20260621 05:21:45.824636 11470 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.32.2:36193
W20260621 05:21:45.904107 11793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:45.904345 11793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:45.904381 11793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:45.905506 11793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:45.905570 11793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:45.905615 11793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:21:45.907006 11793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:43509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:42067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:45.907274 11793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:45.907563 11793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:45.912436 11798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:45.912447 11801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:45.912447 11799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:45.912915 11793 server_base.cc:1034] running on GCE node
I20260621 05:21:45.913261 11793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:45.913545 11793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:45.914700 11793 hybrid_clock.cc:648] HybridClock initialized: now 1782019305914690 us; error 54 us; skew 500 ppm
I20260621 05:21:45.916374 11793 webserver.cc:460] Webserver started at http://127.2.32.3:34111/ using document root <none> and password file <none>
I20260621 05:21:45.916713 11793 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:45.916826 11793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:45.916992 11793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:21:45.918246 11793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "3e48db6734e249149ad13c3358ff216f"
format_stamp: "Formatted at 2026-06-21 05:21:45 on dist-test-slave-b357"
server_key: "092e87038a0ded2ca65a9b404e1caca9"
server_key_iv: "f9ad6a31fc1f8fe56018a49f6d1dc3c6"
server_key_version: "encryptionkey@0"
I20260621 05:21:45.918658 11793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "3e48db6734e249149ad13c3358ff216f"
format_stamp: "Formatted at 2026-06-21 05:21:45 on dist-test-slave-b357"
server_key: "092e87038a0ded2ca65a9b404e1caca9"
server_key_iv: "f9ad6a31fc1f8fe56018a49f6d1dc3c6"
server_key_version: "encryptionkey@0"
I20260621 05:21:45.920192 11793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:21:45.921180 11807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:45.921383 11793 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:45.921468 11793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "3e48db6734e249149ad13c3358ff216f"
format_stamp: "Formatted at 2026-06-21 05:21:45 on dist-test-slave-b357"
server_key: "092e87038a0ded2ca65a9b404e1caca9"
server_key_iv: "f9ad6a31fc1f8fe56018a49f6d1dc3c6"
server_key_version: "encryptionkey@0"
I20260621 05:21:45.921525 11793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:45.937340 11793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:45.937595 11793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:45.937695 11793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:45.937861 11793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:21:45.938163 11793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:21:45.938194 11793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:45.938216 11793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:21:45.938272 11793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:45.943869 11793 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:38891
I20260621 05:21:45.943951 11920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:38891 every 8 connection(s)
I20260621 05:21:45.944245 11793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:21:45.947619 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 11793
I20260621 05:21:45.947768 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:21:45.948172 2176 external_mini_cluster.cc:1428] Setting key 2304ad29a027c7068c70b16a64368683
I20260621 05:21:45.951099 11510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:45.945202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:36793 (local address 127.2.32.62:42067)
0621 05:21:45.945287 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:45.945289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:45.945496 (+ 207us) server_negotiation.cc:407] Connection header received
0621 05:21:45.945942 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:45.945944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:45.945953 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:45.946010 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:45.946763 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.947637 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.949238 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.949466 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.949630 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:45.950629 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:45.950631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:45.950631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:21:45.950636 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:45.950656 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:21:45.950660 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:45.950855 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:45.950969 (+ 114us) server_negotiation.cc:299] Negotiation successful
0621 05:21:45.951005 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 05:21:45.951388 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:45.945116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:42067 (local address 127.2.32.3:36793)
0621 05:21:45.945314 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:45.945347 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:45.945834 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:45.946062 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:45.946064 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:45.946072 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:45.946566 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:45.946570 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.947896 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.947897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:45.949150 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:45.949155 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:45.949640 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:45.949641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:45.949797 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:45.950432 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:21:45.950433 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:45.950464 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:21:45.950491 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:21:45.950857 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:21:45.950865 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:21:45.950869 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:21:45.950973 (+ 104us) client_negotiation.cc:240] Negotiation successful
0621 05:21:45.951159 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260621 05:21:45.952281 11921 heartbeater.cc:347] Connected to a master server at 127.2.32.62:42067
I20260621 05:21:45.952524 11921 heartbeater.cc:464] Registering TS with master...
I20260621 05:21:45.952878 11921 heartbeater.cc:511] Master 127.2.32.62:42067 requested a full tablet report, sending...
I20260621 05:21:45.953464 11470 ts_manager.cc:194] Registered new tserver with Master: 3e48db6734e249149ad13c3358ff216f (127.2.32.3:38891)
I20260621 05:21:45.953886 11470 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.32.3:36793
I20260621 05:21:45.960736 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 05:21:45.962556 11510 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:45.962087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42476 (local address 127.2.32.62:42067)
0621 05:21:45.962149 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:45.962151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:45.962248 (+ 97us) server_negotiation.cc:407] Connection header received
0621 05:21:45.962300 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:45.962301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:45.962308 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:45.962363 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:45.962479 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42476: BlockingRecv error: recv got EOF from 127.0.0.1:42476 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 05:21:45.962682 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 11531
I20260621 05:21:45.968122 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 11662
I20260621 05:21:45.972433 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 11793
I20260621 05:21:45.976476 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 11440
2026-06-21T05:21:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (609 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-21T05:21:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:21:45Z Disabled control of system clock
I20260621 05:21:46.004065 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:44489
--webserver_interface=127.2.32.62
--webserver_port=0
--builtin_ntp_servers=127.2.32.20:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:44489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 05:21:46.085562 11932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:46.085789 11932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:46.085813 11932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:46.086872 11932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:21:46.086908 11932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:46.086921 11932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:46.086933 11932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:21:46.086946 11932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:21:46.088234 11932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:44489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:44489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:46.088447 11932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:46.088680 11932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:46.093567 11940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:46.093657 11938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:46.093657 11937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:46.094058 11932 server_base.cc:1034] running on GCE node
I20260621 05:21:46.094367 11932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:46.094698 11932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:46.095919 11932 hybrid_clock.cc:648] HybridClock initialized: now 1782019306095898 us; error 39 us; skew 500 ppm
I20260621 05:21:46.097504 11932 webserver.cc:460] Webserver started at http://127.2.32.62:45655/ using document root <none> and password file <none>
I20260621 05:21:46.097812 11932 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:46.097898 11932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:46.098047 11932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:21:46.099175 11932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "aaad9c86ac714f809a22512166e0be90"
format_stamp: "Formatted at 2026-06-21 05:21:46 on dist-test-slave-b357"
server_key: "18a9bd822aa76ea23311fc174bc9d553"
server_key_iv: "52ed1db0d34aaef06b381221fcb0e31f"
server_key_version: "encryptionkey@0"
I20260621 05:21:46.099526 11932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "aaad9c86ac714f809a22512166e0be90"
format_stamp: "Formatted at 2026-06-21 05:21:46 on dist-test-slave-b357"
server_key: "18a9bd822aa76ea23311fc174bc9d553"
server_key_iv: "52ed1db0d34aaef06b381221fcb0e31f"
server_key_version: "encryptionkey@0"
I20260621 05:21:46.101200 11932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:21:46.102056 11946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:46.102290 11932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:46.102408 11932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "aaad9c86ac714f809a22512166e0be90"
format_stamp: "Formatted at 2026-06-21 05:21:46 on dist-test-slave-b357"
server_key: "18a9bd822aa76ea23311fc174bc9d553"
server_key_iv: "52ed1db0d34aaef06b381221fcb0e31f"
server_key_version: "encryptionkey@0"
I20260621 05:21:46.102490 11932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:46.105792 11932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:46.106050 11932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:46.106191 11932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:46.110131 11932 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:44489
I20260621 05:21:46.110167 11998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:44489 every 8 connection(s)
I20260621 05:21:46.110466 11932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:21:46.111032 11999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:46.113704 11999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90: Bootstrap starting.
I20260621 05:21:46.114276 11999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:46.114591 11999 log.cc:824] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90: Log is configured to *not* fsync() on all Append() calls
I20260621 05:21:46.115651 11999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90: No bootstrap required, opened a new log
I20260621 05:21:46.117205 11999 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaad9c86ac714f809a22512166e0be90" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 44489 } }
I20260621 05:21:46.117336 11999 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:46.117360 11999 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaad9c86ac714f809a22512166e0be90, State: Initialized, Role: FOLLOWER
I20260621 05:21:46.117503 11999 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaad9c86ac714f809a22512166e0be90" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 44489 } }
I20260621 05:21:46.117576 11999 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:21:46.117604 11999 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:21:46.117655 11999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:46.118224 11999 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaad9c86ac714f809a22512166e0be90" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 44489 } }
I20260621 05:21:46.118312 11999 leader_election.cc:302] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaad9c86ac714f809a22512166e0be90; no voters:
I20260621 05:21:46.118467 11999 leader_election.cc:288] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:21:46.118541 12002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:21:46.118618 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 11932
I20260621 05:21:46.118716 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:21:46.118719 11999 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:21:46.118778 12002 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90 [term 1 LEADER]: Becoming Leader. State: Replica: aaad9c86ac714f809a22512166e0be90, State: Running, Role: LEADER
I20260621 05:21:46.118865 12002 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaad9c86ac714f809a22512166e0be90" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 44489 } }
I20260621 05:21:46.118916 2176 external_mini_cluster.cc:1428] Setting key 328397a8008d4488193bd63d61e3ff79
I20260621 05:21:46.119415 12003 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaad9c86ac714f809a22512166e0be90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaad9c86ac714f809a22512166e0be90" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 44489 } } }
I20260621 05:21:46.119511 12003 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90 [sys.catalog]: This master's current role is: LEADER
I20260621 05:21:46.119674 12004 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaad9c86ac714f809a22512166e0be90. Latest consensus state: current_term: 1 leader_uuid: "aaad9c86ac714f809a22512166e0be90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaad9c86ac714f809a22512166e0be90" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 44489 } } }
I20260621 05:21:46.119742 12013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:21:46.119779 12004 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90 [sys.catalog]: This master's current role is: LEADER
I20260621 05:21:46.120244 12013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:21:46.121843 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:46.119557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46746 (local address 127.2.32.62:44489)
0621 05:21:46.120300 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:46.120308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:46.120344 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:21:46.120461 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:46.120467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:46.120515 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:46.120649 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:46.121351 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:46.121353 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:46.121353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:21:46.121356 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:46.121411 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:21:46.121425 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:46.121494 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:46.121618 (+ 124us) server_negotiation.cc:299] Negotiation successful
0621 05:21:46.121677 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":56,"threads_started":1}
I20260621 05:21:46.121978 12013 catalog_manager.cc:1347] Generated new cluster ID: fa7163c545d34e3bb2c482438e850b8d
I20260621 05:21:46.122028 12013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:21:46.134017 12013 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:21:46.134197 12013 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:21:46.141413 12013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aaad9c86ac714f809a22512166e0be90: Generated new TSK 0
I20260621 05:21:46.141613 12013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:21:46.142952 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:44489
--builtin_ntp_servers=127.2.32.20:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 05:21:46.257963 12023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:46.258315 12023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:46.258351 12023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:46.260171 12023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:46.260247 12023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:46.260289 12023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:21:46.262298 12023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:44489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:46.262570 12023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:46.262936 12023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:46.269222 12029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:46.269366 12028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:46.269222 12031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:46.269520 12023 server_base.cc:1034] running on GCE node
I20260621 05:21:46.269714 12023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:46.269965 12023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:46.271199 12023 hybrid_clock.cc:648] HybridClock initialized: now 1782019306271182 us; error 30 us; skew 500 ppm
I20260621 05:21:46.272625 12023 webserver.cc:460] Webserver started at http://127.2.32.1:35871/ using document root <none> and password file <none>
I20260621 05:21:46.272900 12023 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:46.272976 12023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:46.273125 12023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:21:46.274199 12023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "573c17a2741b410ca96b578e818ec803"
format_stamp: "Formatted at 2026-06-21 05:21:46 on dist-test-slave-b357"
server_key: "0659f2c9cfc8b942d23f3d252cdc2a9e"
server_key_iv: "435ea5b5cecc9813f7d65d4a638d6f4a"
server_key_version: "encryptionkey@0"
I20260621 05:21:46.274574 12023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "573c17a2741b410ca96b578e818ec803"
format_stamp: "Formatted at 2026-06-21 05:21:46 on dist-test-slave-b357"
server_key: "0659f2c9cfc8b942d23f3d252cdc2a9e"
server_key_iv: "435ea5b5cecc9813f7d65d4a638d6f4a"
server_key_version: "encryptionkey@0"
I20260621 05:21:46.276137 12023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:21:46.277091 12037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:46.277330 12023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:21:46.277426 12023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "573c17a2741b410ca96b578e818ec803"
format_stamp: "Formatted at 2026-06-21 05:21:46 on dist-test-slave-b357"
server_key: "0659f2c9cfc8b942d23f3d252cdc2a9e"
server_key_iv: "435ea5b5cecc9813f7d65d4a638d6f4a"
server_key_version: "encryptionkey@0"
I20260621 05:21:46.277498 12023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:46.282203 12023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:46.282490 12023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:46.282621 12023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:46.282835 12023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:21:46.283144 12023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:21:46.283185 12023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:46.283212 12023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:21:46.283236 12023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:46.291666 12023 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:40381
I20260621 05:21:46.291893 12150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:40381 every 8 connection(s)
I20260621 05:21:46.292117 12023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:21:46.295238 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:46.292881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:55753 (local address 127.2.32.62:44489)
0621 05:21:46.292950 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:46.292953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:46.293193 (+ 240us) server_negotiation.cc:407] Connection header received
0621 05:21:46.293688 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:46.293689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:46.293696 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:46.293750 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:46.294939 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:46.294940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:46.294941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:21:46.294945 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:46.294960 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:21:46.294964 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:46.295014 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:46.295116 (+ 102us) server_negotiation.cc:299] Negotiation successful
0621 05:21:46.295170 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 05:21:46.295442 12153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:46.292885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:44489 (local address 127.2.32.1:55753)
0621 05:21:46.293059 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:46.293098 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:46.293586 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:46.293818 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:46.293822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:46.293831 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:46.294788 (+ 957us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:21:46.294788 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:46.294813 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:21:46.294839 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:21:46.295031 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:21:46.295035 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:21:46.295037 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:21:46.295125 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 05:21:46.295233 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260621 05:21:46.295907 12151 heartbeater.cc:347] Connected to a master server at 127.2.32.62:44489
I20260621 05:21:46.296001 12151 heartbeater.cc:464] Registering TS with master...
I20260621 05:21:46.296150 12151 heartbeater.cc:511] Master 127.2.32.62:44489 requested a full tablet report, sending...
I20260621 05:21:46.296566 11963 ts_manager.cc:194] Registered new tserver with Master: 573c17a2741b410ca96b578e818ec803 (127.2.32.1:40381)
I20260621 05:21:46.299263 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 12023
I20260621 05:21:46.299363 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:21:46.299553 2176 external_mini_cluster.cc:1428] Setting key 2c73d8e3e5e29368f815170f06f600b4
I20260621 05:21:46.300424 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:44489
--builtin_ntp_servers=127.2.32.20:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 05:21:46.388592 12154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:46.388832 12154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:46.388866 12154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:46.389986 12154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:46.390048 12154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:46.390093 12154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:21:46.391469 12154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:44489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:46.391721 12154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:46.391996 12154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:46.397387 12162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:46.397401 12160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:46.397419 12159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:46.397418 12154 server_base.cc:1034] running on GCE node
I20260621 05:21:46.397804 12154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:46.398029 12154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:46.399247 12154 hybrid_clock.cc:648] HybridClock initialized: now 1782019306399225 us; error 41 us; skew 500 ppm
I20260621 05:21:46.400749 12154 webserver.cc:460] Webserver started at http://127.2.32.2:43013/ using document root <none> and password file <none>
I20260621 05:21:46.401130 12154 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:46.401216 12154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:46.401388 12154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:21:46.402731 12154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "46f1bd7bac45464581229b82e442397c"
format_stamp: "Formatted at 2026-06-21 05:21:46 on dist-test-slave-b357"
server_key: "4bc5ce59c363443236b9cfa52817ae4e"
server_key_iv: "276ecf57865ea0c6af227bda141626ee"
server_key_version: "encryptionkey@0"
I20260621 05:21:46.403465 12154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "46f1bd7bac45464581229b82e442397c"
format_stamp: "Formatted at 2026-06-21 05:21:46 on dist-test-slave-b357"
server_key: "4bc5ce59c363443236b9cfa52817ae4e"
server_key_iv: "276ecf57865ea0c6af227bda141626ee"
server_key_version: "encryptionkey@0"
I20260621 05:21:46.404958 12154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:21:46.406000 12168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:46.406315 12154 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:46.406407 12154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "46f1bd7bac45464581229b82e442397c"
format_stamp: "Formatted at 2026-06-21 05:21:46 on dist-test-slave-b357"
server_key: "4bc5ce59c363443236b9cfa52817ae4e"
server_key_iv: "276ecf57865ea0c6af227bda141626ee"
server_key_version: "encryptionkey@0"
I20260621 05:21:46.406472 12154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:46.409847 12154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:46.410079 12154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:46.410192 12154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:46.410372 12154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:21:46.410624 12154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:21:46.410670 12154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:46.410693 12154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:21:46.410708 12154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:46.417049 12154 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:36409
I20260621 05:21:46.417122 12281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:36409 every 8 connection(s)
I20260621 05:21:46.417423 12154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:21:46.420893 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:46.418146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:48441 (local address 127.2.32.62:44489)
0621 05:21:46.418199 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:46.418201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:46.418461 (+ 260us) server_negotiation.cc:407] Connection header received
0621 05:21:46.419207 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:46.419209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:46.419220 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:46.419313 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:46.420625 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:46.420627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:46.420627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:21:46.420632 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:46.420653 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:21:46.420658 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:46.420717 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:46.420796 (+ 79us) server_negotiation.cc:299] Negotiation successful
0621 05:21:46.420810 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:21:46.421038 12284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:46.418154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:44489 (local address 127.2.32.2:48441)
0621 05:21:46.418346 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:46.418376 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:46.418948 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:46.419383 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:46.419386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:46.419394 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:46.420445 (+ 1051us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:21:46.420445 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:46.420470 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:21:46.420496 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:21:46.420725 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:21:46.420728 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:21:46.420730 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:21:46.420789 (+ 59us) client_negotiation.cc:240] Negotiation successful
0621 05:21:46.420833 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
I20260621 05:21:46.421599 12282 heartbeater.cc:347] Connected to a master server at 127.2.32.62:44489
I20260621 05:21:46.421702 12282 heartbeater.cc:464] Registering TS with master...
I20260621 05:21:46.421847 12282 heartbeater.cc:511] Master 127.2.32.62:44489 requested a full tablet report, sending...
I20260621 05:21:46.422506 11963 ts_manager.cc:194] Registered new tserver with Master: 46f1bd7bac45464581229b82e442397c (127.2.32.2:36409)
I20260621 05:21:46.426308 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 12154
I20260621 05:21:46.426405 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:21:46.426613 2176 external_mini_cluster.cc:1428] Setting key 61efe473e9496e181c93e58f023d8464
I20260621 05:21:46.427866 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:44489
--builtin_ntp_servers=127.2.32.20:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 05:21:46.513948 12285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:46.514241 12285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:46.514279 12285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:46.516028 12285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:46.516081 12285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:46.516121 12285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:21:46.517516 12285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:44489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:46.517711 12285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:46.517941 12285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:46.523011 12293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:46.522986 12291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:46.523036 12290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:46.523340 12285 server_base.cc:1034] running on GCE node
I20260621 05:21:46.523555 12285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:46.523780 12285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:46.524955 12285 hybrid_clock.cc:648] HybridClock initialized: now 1782019306524953 us; error 40 us; skew 500 ppm
I20260621 05:21:46.526620 12285 webserver.cc:460] Webserver started at http://127.2.32.3:37801/ using document root <none> and password file <none>
I20260621 05:21:46.526960 12285 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:46.527038 12285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:46.527256 12285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:21:46.528517 12285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "24a93730ec3941be84a57f72ca0d7a63"
format_stamp: "Formatted at 2026-06-21 05:21:46 on dist-test-slave-b357"
server_key: "681be8959f07c38305b1479c1178b5d7"
server_key_iv: "9d0eab793e732c66c42a668eb05d4aca"
server_key_version: "encryptionkey@0"
I20260621 05:21:46.528924 12285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "24a93730ec3941be84a57f72ca0d7a63"
format_stamp: "Formatted at 2026-06-21 05:21:46 on dist-test-slave-b357"
server_key: "681be8959f07c38305b1479c1178b5d7"
server_key_iv: "9d0eab793e732c66c42a668eb05d4aca"
server_key_version: "encryptionkey@0"
I20260621 05:21:46.530762 12285 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260621 05:21:46.531908 12299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:46.532196 12285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:21:46.532290 12285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "24a93730ec3941be84a57f72ca0d7a63"
format_stamp: "Formatted at 2026-06-21 05:21:46 on dist-test-slave-b357"
server_key: "681be8959f07c38305b1479c1178b5d7"
server_key_iv: "9d0eab793e732c66c42a668eb05d4aca"
server_key_version: "encryptionkey@0"
I20260621 05:21:46.532392 12285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:46.536197 12285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:46.536504 12285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:46.536654 12285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:46.536872 12285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:21:46.537221 12285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:21:46.537267 12285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:46.537290 12285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:21:46.537303 12285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:46.543771 12285 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:46593
I20260621 05:21:46.543838 12412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:46593 every 8 connection(s)
I20260621 05:21:46.544169 12285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:21:46.547490 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:46.544865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:59037 (local address 127.2.32.62:44489)
0621 05:21:46.544940 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:46.544943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:46.545303 (+ 360us) server_negotiation.cc:407] Connection header received
0621 05:21:46.546066 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:46.546068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:46.546073 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:46.546143 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:46.547298 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:46.547300 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:46.547300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:21:46.547304 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:46.547317 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:21:46.547321 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:46.547374 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:46.547424 (+ 50us) server_negotiation.cc:299] Negotiation successful
0621 05:21:46.547437 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 05:21:46.547698 12415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:46.544829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:44489 (local address 127.2.32.3:59037)
0621 05:21:46.545113 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:46.545159 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:46.545889 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:46.546201 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:46.546204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:46.546214 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:46.547103 (+ 889us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:21:46.547103 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:46.547197 (+ 94us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:21:46.547226 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:21:46.547370 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:21:46.547372 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:21:46.547374 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:21:46.547425 (+ 51us) client_negotiation.cc:240] Negotiation successful
0621 05:21:46.547469 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":105,"threads_started":1}
I20260621 05:21:46.548420 12413 heartbeater.cc:347] Connected to a master server at 127.2.32.62:44489
I20260621 05:21:46.548507 12413 heartbeater.cc:464] Registering TS with master...
I20260621 05:21:46.548612 12413 heartbeater.cc:511] Master 127.2.32.62:44489 requested a full tablet report, sending...
I20260621 05:21:46.549008 11963 ts_manager.cc:194] Registered new tserver with Master: 24a93730ec3941be84a57f72ca0d7a63 (127.2.32.3:46593)
I20260621 05:21:46.553153 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 12285
I20260621 05:21:46.553272 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:21:46.553452 2176 external_mini_cluster.cc:1428] Setting key 4231c2bfb52de9a92f9b6db63b529ffd
I20260621 05:21:46.554335 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 05:21:46.556038 12012 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:21:46.555609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46758 (local address 127.2.32.62:44489)
0621 05:21:46.555644 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:46.555646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:46.555814 (+ 168us) server_negotiation.cc:407] Connection header received
0621 05:21:46.555834 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:46.555835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:46.555839 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:46.555877 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:46.555984 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46758: BlockingRecv error: recv got EOF from 127.0.0.1:46758 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:21:46.556090 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 12023
I20260621 05:21:46.561779 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 12154
I20260621 05:21:46.566561 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 12285
I20260621 05:21:46.570746 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 11932
2026-06-21T05:21:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (594 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:21:46 dist-test-slave-b357 krb5kdc[12423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:21:46 dist-test-slave-b357 krb5kdc[12423](info): set up 2 sockets
Jun 21 05:21:46 dist-test-slave-b357 krb5kdc[12423](info): commencing operation
krb5kdc: starting...
W20260621 05:21:47.632149 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:21:47 dist-test-slave-b357 krb5kdc[12423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:21:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:21:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:21:47.757984 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:40879
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:40879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:21:47.843887 12436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:47.844128 12436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:47.844175 12436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:47.845770 12436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:21:47.845829 12436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:47.845853 12436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:47.845881 12436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:21:47.845911 12436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:21:47.847553 12436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:40879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:40879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:47.847824 12436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:47.848060 12436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:47.853053 12441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:47.853039 12442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:47.853042 12444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:47.853116 12436 server_base.cc:1034] running on GCE node
I20260621 05:21:47.853431 12436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:47.853668 12436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:47.854820 12436 hybrid_clock.cc:648] HybridClock initialized: now 1782019307854796 us; error 49 us; skew 500 ppm
Jun 21 05:21:47 dist-test-slave-b357 krb5kdc[12423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019307, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:21:47.858162 12436 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:21:47.858609 12436 webserver.cc:460] Webserver started at http://127.2.32.62:35651/ using document root <none> and password file <none>
I20260621 05:21:47.858845 12436 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:47.858893 12436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:47.859009 12436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:21:47.860088 12436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "e9bb75f09ebc4cdfb5ba8675de624706"
format_stamp: "Formatted at 2026-06-21 05:21:47 on dist-test-slave-b357"
server_key: "96a007b45bac28d4e2287bb0e673d1d3"
server_key_iv: "46bcb20c3651447bd12a411436696667"
server_key_version: "encryptionkey@0"
I20260621 05:21:47.860435 12436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "e9bb75f09ebc4cdfb5ba8675de624706"
format_stamp: "Formatted at 2026-06-21 05:21:47 on dist-test-slave-b357"
server_key: "96a007b45bac28d4e2287bb0e673d1d3"
server_key_iv: "46bcb20c3651447bd12a411436696667"
server_key_version: "encryptionkey@0"
I20260621 05:21:47.861744 12436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:21:47.862661 12451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:47.862870 12436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:47.862952 12436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "e9bb75f09ebc4cdfb5ba8675de624706"
format_stamp: "Formatted at 2026-06-21 05:21:47 on dist-test-slave-b357"
server_key: "96a007b45bac28d4e2287bb0e673d1d3"
server_key_iv: "46bcb20c3651447bd12a411436696667"
server_key_version: "encryptionkey@0"
I20260621 05:21:47.863005 12436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:47.876448 12436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:47.879117 12436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:47.879338 12436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:47.883001 12436 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:40879
I20260621 05:21:47.883100 12503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:40879 every 8 connection(s)
I20260621 05:21:47.883508 12436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:21:47.884795 12504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:47.887931 12504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706: Bootstrap starting.
I20260621 05:21:47.888672 12504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:47.889017 12504 log.cc:824] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706: Log is configured to *not* fsync() on all Append() calls
I20260621 05:21:47.890043 12504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706: No bootstrap required, opened a new log
I20260621 05:21:47.892012 12504 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bb75f09ebc4cdfb5ba8675de624706" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40879 } }
I20260621 05:21:47.892179 12504 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:47.892205 12504 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9bb75f09ebc4cdfb5ba8675de624706, State: Initialized, Role: FOLLOWER
I20260621 05:21:47.892376 12504 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bb75f09ebc4cdfb5ba8675de624706" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40879 } }
I20260621 05:21:47.892473 12504 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:21:47.892503 12504 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:21:47.892555 12504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:47.893554 12504 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bb75f09ebc4cdfb5ba8675de624706" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40879 } }
I20260621 05:21:47.893689 12504 leader_election.cc:302] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9bb75f09ebc4cdfb5ba8675de624706; no voters:
I20260621 05:21:47.893798 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 12436
I20260621 05:21:47.893927 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:21:47.893921 12504 leader_election.cc:288] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:21:47.893999 12507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:21:47.894197 2176 external_mini_cluster.cc:1428] Setting key bc8a2d9e718602fec802519acc59fbf9
I20260621 05:21:47.894228 12504 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:21:47.894279 12507 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706 [term 1 LEADER]: Becoming Leader. State: Replica: e9bb75f09ebc4cdfb5ba8675de624706, State: Running, Role: LEADER
I20260621 05:21:47.894392 12507 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bb75f09ebc4cdfb5ba8675de624706" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40879 } }
I20260621 05:21:47.894945 12508 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9bb75f09ebc4cdfb5ba8675de624706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bb75f09ebc4cdfb5ba8675de624706" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40879 } } }
I20260621 05:21:47.895059 12508 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706 [sys.catalog]: This master's current role is: LEADER
I20260621 05:21:47.895299 12509 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9bb75f09ebc4cdfb5ba8675de624706. Latest consensus state: current_term: 1 leader_uuid: "e9bb75f09ebc4cdfb5ba8675de624706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bb75f09ebc4cdfb5ba8675de624706" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40879 } } }
I20260621 05:21:47.895390 12509 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706 [sys.catalog]: This master's current role is: LEADER
I20260621 05:21:47.895654 12518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:21:47.896236 12518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:21:47.898113 12518 catalog_manager.cc:1347] Generated new cluster ID: 6f355371de25402eb083530411ce2cdb
I20260621 05:21:47.898190 12518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:21:47 dist-test-slave-b357 krb5kdc[12423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:21:47.904407 12518 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:21:47.905233 12518 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:21:47.907665 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:47.894989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.2.32.62:40879)
0621 05:21:47.895285 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:47.895296 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:47.895331 (+ 35us) server_negotiation.cc:407] Connection header received
0621 05:21:47.895476 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:47.895485 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:47.895553 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:47.895715 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:47.896808 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:47.898049 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:47.899360 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:47.899973 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:47.900092 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:47.903043 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:47.903047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:47.903047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:21:47.903060 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:47.905945 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:47.906389 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:47.906392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:47.906392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:47.906511 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:47.906722 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:47.906723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:47.906724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:47.907011 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:47.907210 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:47.907305 (+ 95us) server_negotiation.cc:299] Negotiation successful
0621 05:21:47.907455 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":142,"threads_started":1}
I20260621 05:21:47.913781 12518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706: Generated new TSK 0
I20260621 05:21:47.913951 12518 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
I20260621 05:21:47.945547 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:40879
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:21:48.030671 12530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:48.030931 12530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:48.030961 12530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:48.032409 12530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:48.032469 12530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:48.032500 12530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:21:48.034081 12530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:40879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:48.034368 12530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:48.034667 12530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:48.040221 12536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:48.040305 12535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:48.040309 12538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:48.040513 12530 server_base.cc:1034] running on GCE node
I20260621 05:21:48.040758 12530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:48.041047 12530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:48.042239 12530 hybrid_clock.cc:648] HybridClock initialized: now 1782019308042217 us; error 40 us; skew 500 ppm
Jun 21 05:21:48 dist-test-slave-b357 krb5kdc[12423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019308, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:21:48.045069 12530 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:21:48.045629 12530 webserver.cc:460] Webserver started at http://127.2.32.1:45131/ using document root <none> and password file <none>
I20260621 05:21:48.045872 12530 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:48.045938 12530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:48.046070 12530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:21:48.047542 12530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "231674478bae4ce5ac9fc3354a412421"
format_stamp: "Formatted at 2026-06-21 05:21:48 on dist-test-slave-b357"
server_key: "940b797fb5ffc3bcd19d856fa1f6d699"
server_key_iv: "7c35882c249096416f2d1902bed17494"
server_key_version: "encryptionkey@0"
I20260621 05:21:48.048010 12530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "231674478bae4ce5ac9fc3354a412421"
format_stamp: "Formatted at 2026-06-21 05:21:48 on dist-test-slave-b357"
server_key: "940b797fb5ffc3bcd19d856fa1f6d699"
server_key_iv: "7c35882c249096416f2d1902bed17494"
server_key_version: "encryptionkey@0"
I20260621 05:21:48.049779 12530 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 05:21:48.050592 12545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:48.050778 12530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:48.050885 12530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "231674478bae4ce5ac9fc3354a412421"
format_stamp: "Formatted at 2026-06-21 05:21:48 on dist-test-slave-b357"
server_key: "940b797fb5ffc3bcd19d856fa1f6d699"
server_key_iv: "7c35882c249096416f2d1902bed17494"
server_key_version: "encryptionkey@0"
I20260621 05:21:48.050980 12530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:48.062748 12530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:48.064841 12530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:48.065027 12530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:48.065239 12530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:21:48.065539 12530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:21:48.065598 12530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:48.065637 12530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:21:48.065663 12530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:48.071593 12530 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:36993
I20260621 05:21:48.071645 12658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:36993 every 8 connection(s)
I20260621 05:21:48.071969 12530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:21:48 dist-test-slave-b357 krb5kdc[12423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019308, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:21:48.080633 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 12530
I20260621 05:21:48.080775 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:21:48.081100 2176 external_mini_cluster.cc:1428] Setting key be2153559fd5e996fbb7af458bdcfcb3
I20260621 05:21:48.081794 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:48.072737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:33929 (local address 127.2.32.62:40879)
0621 05:21:48.072825 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:48.072833 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:48.072997 (+ 164us) server_negotiation.cc:407] Connection header received
0621 05:21:48.073546 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:48.073547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:48.073563 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:48.073616 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:48.074520 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.075028 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.075999 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.076197 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.076279 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:48.078714 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:48.078716 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:48.078717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:21:48.078724 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:48.080359 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:48.080789 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:48.080791 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:48.080791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:48.080854 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:48.081076 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:48.081077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:48.081078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:48.081230 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:48.081307 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:48.081610 (+ 303us) server_negotiation.cc:299] Negotiation successful
0621 05:21:48.081705 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 05:21:48.082049 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:48.072650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40879 (local address 127.2.32.1:33929)
0621 05:21:48.072860 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:48.072907 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:48.073418 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:48.073679 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:48.073683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:48.074239 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:48.074422 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:48.074426 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.075118 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.075119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:48.075910 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:48.075912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.076285 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.076286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:48.076402 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:48.076881 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:48.076888 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:48.078622 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:21:48.080477 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:48.080483 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:48.080485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:48.080693 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:48.080916 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:48.080916 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:48.080917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:48.081012 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:48.081299 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:21:48.081304 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:21:48.081465 (+ 161us) client_negotiation.cc:769] Sending connection context
0621 05:21:48.081584 (+ 119us) client_negotiation.cc:240] Negotiation successful
0621 05:21:48.081742 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":103,"threads_started":1}
I20260621 05:21:48.082494 12659 heartbeater.cc:347] Connected to a master server at 127.2.32.62:40879
I20260621 05:21:48.082578 12659 heartbeater.cc:464] Registering TS with master...
I20260621 05:21:48.082742 12659 heartbeater.cc:511] Master 127.2.32.62:40879 requested a full tablet report, sending...
I20260621 05:21:48.083451 12468 ts_manager.cc:194] Registered new tserver with Master: 231674478bae4ce5ac9fc3354a412421 (127.2.32.1:36993)
I20260621 05:21:48.084043 12468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:33929
WARNING: no policy specified for kudu/127.2.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
I20260621 05:21:48.106456 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:40879
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:21:48.192103 12664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:48.192297 12664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:48.192322 12664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:48.193455 12664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:48.193500 12664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:48.193528 12664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:21:48.194964 12664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:40879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:48.195155 12664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:48.195363 12664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:21:48.200248 12664 server_base.cc:1034] running on GCE node
W20260621 05:21:48.200342 12670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:48.200665 12672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:48.200685 12669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:48.200938 12664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:48.201166 12664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:48.202342 12664 hybrid_clock.cc:648] HybridClock initialized: now 1782019308202325 us; error 36 us; skew 500 ppm
Jun 21 05:21:48 dist-test-slave-b357 krb5kdc[12423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019308, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:21:48.208045 12664 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:21:48.208477 12664 webserver.cc:460] Webserver started at http://127.2.32.2:37597/ using document root <none> and password file <none>
I20260621 05:21:48.208688 12664 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:48.208732 12664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:48.208841 12664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:21:48.209859 12664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "b331453ac17e4788a59a374767e4869c"
format_stamp: "Formatted at 2026-06-21 05:21:48 on dist-test-slave-b357"
server_key: "20d24d22422bcad98f0ebb16e6214fdc"
server_key_iv: "4680a4d2a998e8106cfed168ff46a3f4"
server_key_version: "encryptionkey@0"
I20260621 05:21:48.210213 12664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "b331453ac17e4788a59a374767e4869c"
format_stamp: "Formatted at 2026-06-21 05:21:48 on dist-test-slave-b357"
server_key: "20d24d22422bcad98f0ebb16e6214fdc"
server_key_iv: "4680a4d2a998e8106cfed168ff46a3f4"
server_key_version: "encryptionkey@0"
I20260621 05:21:48.211566 12664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:21:48.212370 12679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:48.212544 12664 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:21:48.212630 12664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "b331453ac17e4788a59a374767e4869c"
format_stamp: "Formatted at 2026-06-21 05:21:48 on dist-test-slave-b357"
server_key: "20d24d22422bcad98f0ebb16e6214fdc"
server_key_iv: "4680a4d2a998e8106cfed168ff46a3f4"
server_key_version: "encryptionkey@0"
I20260621 05:21:48.212680 12664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:48.229586 12664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:48.231768 12664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:48.231881 12664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:48.232069 12664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:21:48.232357 12664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:21:48.232394 12664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:48.232416 12664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:21:48.232429 12664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:48.238916 12664 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:36713
I20260621 05:21:48.238976 12792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:36713 every 8 connection(s)
I20260621 05:21:48.239365 12664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:21:48.241343 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 12664
I20260621 05:21:48.241451 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:21:48.241623 2176 external_mini_cluster.cc:1428] Setting key 0af867086801e0f3a524913ccc0b65f6
Jun 21 05:21:48 dist-test-slave-b357 krb5kdc[12423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019308, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.32.3@KRBTEST.COM; defaulting to no policy
I20260621 05:21:48.250388 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:48.240587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:52611 (local address 127.2.32.62:40879)
0621 05:21:48.240670 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:48.240672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:48.240688 (+ 16us) server_negotiation.cc:407] Connection header received
0621 05:21:48.241221 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:48.241222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:48.241236 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:48.241331 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:48.242408 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.243187 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.244491 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.244834 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.244915 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:48.247360 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:48.247362 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:48.247363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:21:48.247370 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:48.249008 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:48.249424 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:48.249425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:48.249426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:48.249482 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:48.249769 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:48.249770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:48.249770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:48.249919 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:48.249995 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:48.250252 (+ 257us) server_negotiation.cc:299] Negotiation successful
0621 05:21:48.250326 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 05:21:48.250523 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:48.240365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40879 (local address 127.2.32.2:52611)
0621 05:21:48.240560 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:48.240607 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:48.241107 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:48.241388 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:48.241392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:48.242094 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:48.242340 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:48.242345 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.243306 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.243307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:48.244346 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:48.244349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.244948 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.244949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:48.245073 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:48.245612 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:48.245621 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:48.247273 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:21:48.249129 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:48.249134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:48.249136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:48.249343 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:48.249578 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:48.249579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:48.249580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:48.249702 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:48.250010 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:21:48.250015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:21:48.250125 (+ 110us) client_negotiation.cc:769] Sending connection context
0621 05:21:48.250208 (+ 83us) client_negotiation.cc:240] Negotiation successful
0621 05:21:48.250326 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":127,"threads_started":1}
I20260621 05:21:48.251081 12793 heartbeater.cc:347] Connected to a master server at 127.2.32.62:40879
I20260621 05:21:48.251225 12793 heartbeater.cc:464] Registering TS with master...
I20260621 05:21:48.251406 12793 heartbeater.cc:511] Master 127.2.32.62:40879 requested a full tablet report, sending...
I20260621 05:21:48.251786 12468 ts_manager.cc:194] Registered new tserver with Master: b331453ac17e4788a59a374767e4869c (127.2.32.2:36713)
I20260621 05:21:48.252221 12468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:52611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
I20260621 05:21:48.269596 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:40879
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:21:48.359082 12798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:21:48.359308 12798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:21:48.359330 12798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:21:48.360672 12798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:21:48.360735 12798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:21:48.360764 12798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:21:48.362087 12798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:40879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:21:48.362262 12798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:21:48.362421 12798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:21:48.368064 12804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:21:48.368034 12803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:48.368037 12798 server_base.cc:1034] running on GCE node
W20260621 05:21:48.368162 12806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:21:48.368635 12798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:21:48.368871 12798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:21:48.370054 12798 hybrid_clock.cc:648] HybridClock initialized: now 1782019308370034 us; error 39 us; skew 500 ppm
Jun 21 05:21:48 dist-test-slave-b357 krb5kdc[12423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019308, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:21:48.372896 12798 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:21:48.373343 12798 webserver.cc:460] Webserver started at http://127.2.32.3:44397/ using document root <none> and password file <none>
I20260621 05:21:48.373565 12798 fs_manager.cc:359] Metadata directory not provided
I20260621 05:21:48.373628 12798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:21:48.373757 12798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:21:48.374744 12798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "417c43734595436583e25fd0e453528f"
format_stamp: "Formatted at 2026-06-21 05:21:48 on dist-test-slave-b357"
server_key: "8c3db472b5e76926fb16423f6e639317"
server_key_iv: "056529c123d9c19f330b76d51561b2de"
server_key_version: "encryptionkey@0"
I20260621 05:21:48.375111 12798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "417c43734595436583e25fd0e453528f"
format_stamp: "Formatted at 2026-06-21 05:21:48 on dist-test-slave-b357"
server_key: "8c3db472b5e76926fb16423f6e639317"
server_key_iv: "056529c123d9c19f330b76d51561b2de"
server_key_version: "encryptionkey@0"
I20260621 05:21:48.376461 12798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:48.377408 12813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:48.377609 12798 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:21:48.377712 12798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "417c43734595436583e25fd0e453528f"
format_stamp: "Formatted at 2026-06-21 05:21:48 on dist-test-slave-b357"
server_key: "8c3db472b5e76926fb16423f6e639317"
server_key_iv: "056529c123d9c19f330b76d51561b2de"
server_key_version: "encryptionkey@0"
I20260621 05:21:48.377795 12798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:21:48.388976 12798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:21:48.391263 12798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:21:48.391417 12798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:21:48.391642 12798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:21:48.391923 12798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:21:48.391973 12798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:48.392014 12798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:21:48.392040 12798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:21:48.398680 12798 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:41015
I20260621 05:21:48.398777 12926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:41015 every 8 connection(s)
I20260621 05:21:48.399195 12798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:21:48.404597 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 12798
I20260621 05:21:48.404757 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:21:48.405009 2176 external_mini_cluster.cc:1428] Setting key a6179e589fcd430cd13c68154449b93d
Jun 21 05:21:48 dist-test-slave-b357 krb5kdc[12423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019308, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:21:48.410238 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:48.400008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:38853 (local address 127.2.32.62:40879)
0621 05:21:48.400089 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:48.400091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:48.400270 (+ 179us) server_negotiation.cc:407] Connection header received
0621 05:21:48.401229 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:48.401231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:48.401240 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:48.401306 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:48.401891 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.402406 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.403498 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.403725 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.403838 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:48.406818 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:48.406819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:48.406819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:21:48.406826 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:48.408718 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:48.409232 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:48.409233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:48.409234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:48.409305 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:48.409615 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:48.409616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:48.409616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:48.409750 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:48.409832 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:48.410096 (+ 264us) server_negotiation.cc:299] Negotiation successful
0621 05:21:48.410172 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 05:21:48.410512 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:48.399983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40879 (local address 127.2.32.3:38853)
0621 05:21:48.400146 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:48.400186 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:48.401132 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:48.401357 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:48.401361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:48.401608 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:21:48.401798 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:48.401802 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.402507 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.402508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:48.403363 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:48.403365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.403873 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.403875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:48.404029 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:48.404866 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:21:48.404877 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:21:48.406710 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:21:48.408862 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:48.408868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:48.408870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:48.409122 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:48.409401 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:21:48.409402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:21:48.409402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:21:48.409511 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:21:48.409821 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:21:48.409825 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:21:48.409952 (+ 127us) client_negotiation.cc:769] Sending connection context
0621 05:21:48.410042 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 05:21:48.410173 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260621 05:21:48.411024 12927 heartbeater.cc:347] Connected to a master server at 127.2.32.62:40879
I20260621 05:21:48.411176 12927 heartbeater.cc:464] Registering TS with master...
I20260621 05:21:48.411351 12927 heartbeater.cc:511] Master 127.2.32.62:40879 requested a full tablet report, sending...
I20260621 05:21:48.411703 12468 ts_manager.cc:194] Registered new tserver with Master: 417c43734595436583e25fd0e453528f (127.2.32.3:41015)
I20260621 05:21:48.412043 12468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:38853
I20260621 05:21:48.416836 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:21:48.425213 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:48.418201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.2.32.62:40879)
0621 05:21:48.418306 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:48.418309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:48.418441 (+ 132us) server_negotiation.cc:407] Connection header received
0621 05:21:48.418465 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:48.418466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:48.418474 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:48.418530 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:21:48.419081 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.419712 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.420677 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.420910 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.420982 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:48.422295 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:21:48.422296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:21:48.422296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:21:48.422302 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:21:48.423806 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:48.424171 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:48.424171 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:48.424172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:48.424221 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:21:48.424495 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:21:48.424496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:21:48.424497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:21:48.424643 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:21:48.424741 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:48.424931 (+ 190us) server_negotiation.cc:299] Negotiation successful
0621 05:21:48.425069 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 05:21:48.427002 12468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:21:48.428653 12468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:21:48.436542 12939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:48.431413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:36993 (local address 127.0.0.1:51026)
0621 05:21:48.431793 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:48.431822 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:48.431902 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:48.432187 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:48.432190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:48.432195 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:21:48.432287 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:48.432288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.433602 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.433603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:48.434829 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:48.434834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.436162 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.436163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:48.436261 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:48.436284 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 05:21:48.436341 (+ 57us) client_negotiation.cc:240] Negotiation successful
0621 05:21:48.436419 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":111,"thread_start_us":80,"threads_started":1}
I20260621 05:21:48.436852 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:48.431485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46242 (local address 127.2.32.3:41015)
0621 05:21:48.431631 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:48.431635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:48.431906 (+ 271us) server_negotiation.cc:407] Connection header received
0621 05:21:48.432023 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:48.432025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:48.432042 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:48.432092 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:21:48.432373 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.433504 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.435485 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.436170 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.436254 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:48.436317 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:48.436557 (+ 240us) server_negotiation.cc:299] Negotiation successful
0621 05:21:48.436741 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260621 05:21:48.436945 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:48.431601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51026 (local address 127.2.32.1:36993)
0621 05:21:48.431790 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:48.431794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:48.431892 (+ 98us) server_negotiation.cc:407] Connection header received
0621 05:21:48.431997 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:48.432000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:48.432016 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:48.432378 (+ 362us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:21:48.432449 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.433494 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.435203 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.436014 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.436697 (+ 683us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:48.436773 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:48.436802 (+ 29us) server_negotiation.cc:299] Negotiation successful
0621 05:21:48.436853 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
I20260621 05:21:48.437366 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:48.431413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:41015 (local address 127.0.0.1:46242)
0621 05:21:48.431776 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:48.431817 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:48.431935 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:48.432187 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:48.432190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:48.432192 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:21:48.432286 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:48.432288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.434524 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.434524 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:48.435392 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:48.435396 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.436363 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.436365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:48.436440 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:48.436446 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:21:48.437161 (+ 715us) client_negotiation.cc:240] Negotiation successful
0621 05:21:48.437243 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":22,"thread_start_us":52,"threads_started":1}
I20260621 05:21:48.437511 12938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:48.431632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:36713 (local address 127.0.0.1:57596)
0621 05:21:48.432144 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:48.432155 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:48.432232 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:48.432500 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:48.432501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:48.432503 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:21:48.432543 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:48.432544 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.433444 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.433445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:48.434789 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:48.434791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.436234 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.436235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:48.436287 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:48.436294 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 05:21:48.437354 (+ 1060us) client_negotiation.cc:240] Negotiation successful
0621 05:21:48.437410 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":5,"thread_start_us":75,"threads_started":1}
I20260621 05:21:48.437762 12943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:48.431866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57596 (local address 127.2.32.2:36713)
0621 05:21:48.432075 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:48.432092 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:48.432205 (+ 113us) server_negotiation.cc:407] Connection header received
0621 05:21:48.432336 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:48.432338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:48.432354 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:48.432411 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:21:48.432636 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.433300 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.435065 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.436046 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.436128 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:48.436200 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:48.437601 (+ 1401us) server_negotiation.cc:299] Negotiation successful
0621 05:21:48.437672 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":89,"threads_started":1}
I20260621 05:21:48.438365 12593 tablet_service.cc:1467] Processing CreateTablet for tablet b4556d755b5a44cdb0b147bf90d1ed7a (DEFAULT_TABLE table=test-table [id=323912eb64b544fa8544c2d4c68b3f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:21:48.438376 12861 tablet_service.cc:1467] Processing CreateTablet for tablet b4556d755b5a44cdb0b147bf90d1ed7a (DEFAULT_TABLE table=test-table [id=323912eb64b544fa8544c2d4c68b3f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:21:48.438455 12727 tablet_service.cc:1467] Processing CreateTablet for tablet b4556d755b5a44cdb0b147bf90d1ed7a (DEFAULT_TABLE table=test-table [id=323912eb64b544fa8544c2d4c68b3f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:21:48.438784 12861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4556d755b5a44cdb0b147bf90d1ed7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:48.438784 12593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4556d755b5a44cdb0b147bf90d1ed7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:48.438808 12727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4556d755b5a44cdb0b147bf90d1ed7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:21:48.442107 12944 tablet_bootstrap.cc:492] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c: Bootstrap starting.
I20260621 05:21:48.442631 12945 tablet_bootstrap.cc:492] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f: Bootstrap starting.
I20260621 05:21:48.443213 12944 tablet_bootstrap.cc:654] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:48.443286 12946 tablet_bootstrap.cc:492] T b4556d755b5a44cdb0b147bf90d1ed7a P 231674478bae4ce5ac9fc3354a412421: Bootstrap starting.
I20260621 05:21:48.443508 12945 tablet_bootstrap.cc:654] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:48.443840 12944 log.cc:824] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c: Log is configured to *not* fsync() on all Append() calls
I20260621 05:21:48.443946 12946 tablet_bootstrap.cc:654] T b4556d755b5a44cdb0b147bf90d1ed7a P 231674478bae4ce5ac9fc3354a412421: Neither blocks nor log segments found. Creating new log.
I20260621 05:21:48.444109 12945 log.cc:824] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f: Log is configured to *not* fsync() on all Append() calls
I20260621 05:21:48.444499 12946 log.cc:824] T b4556d755b5a44cdb0b147bf90d1ed7a P 231674478bae4ce5ac9fc3354a412421: Log is configured to *not* fsync() on all Append() calls
I20260621 05:21:48.444727 12944 tablet_bootstrap.cc:492] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c: No bootstrap required, opened a new log
I20260621 05:21:48.444813 12944 ts_tablet_manager.cc:1397] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:21:48.444958 12945 tablet_bootstrap.cc:492] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f: No bootstrap required, opened a new log
I20260621 05:21:48.445039 12945 ts_tablet_manager.cc:1397] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:21:48.445360 12946 tablet_bootstrap.cc:492] T b4556d755b5a44cdb0b147bf90d1ed7a P 231674478bae4ce5ac9fc3354a412421: No bootstrap required, opened a new log
I20260621 05:21:48.445466 12946 ts_tablet_manager.cc:1397] T b4556d755b5a44cdb0b147bf90d1ed7a P 231674478bae4ce5ac9fc3354a412421: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:21:48.446591 12944 raft_consensus.cc:348] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c43734595436583e25fd0e453528f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41015 } } peers { permanent_uuid: "231674478bae4ce5ac9fc3354a412421" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36993 } } peers { permanent_uuid: "b331453ac17e4788a59a374767e4869c" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 36713 } }
I20260621 05:21:48.446735 12944 raft_consensus.cc:374] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:48.446758 12944 raft_consensus.cc:729] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b331453ac17e4788a59a374767e4869c, State: Initialized, Role: FOLLOWER
I20260621 05:21:48.446872 12944 consensus_queue.cc:260] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c43734595436583e25fd0e453528f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41015 } } peers { permanent_uuid: "231674478bae4ce5ac9fc3354a412421" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36993 } } peers { permanent_uuid: "b331453ac17e4788a59a374767e4869c" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 36713 } }
I20260621 05:21:48.446980 12945 raft_consensus.cc:348] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c43734595436583e25fd0e453528f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41015 } } peers { permanent_uuid: "231674478bae4ce5ac9fc3354a412421" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36993 } } peers { permanent_uuid: "b331453ac17e4788a59a374767e4869c" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 36713 } }
I20260621 05:21:48.447163 12944 ts_tablet_manager.cc:1428] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:21:48.447156 12945 raft_consensus.cc:374] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:48.447184 12793 heartbeater.cc:503] Master 127.2.32.62:40879 was elected leader, sending a full tablet report...
I20260621 05:21:48.447216 12945 raft_consensus.cc:729] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417c43734595436583e25fd0e453528f, State: Initialized, Role: FOLLOWER
I20260621 05:21:48.447347 12945 consensus_queue.cc:260] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c43734595436583e25fd0e453528f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41015 } } peers { permanent_uuid: "231674478bae4ce5ac9fc3354a412421" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36993 } } peers { permanent_uuid: "b331453ac17e4788a59a374767e4869c" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 36713 } }
I20260621 05:21:48.447618 12945 ts_tablet_manager.cc:1428] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260621 05:21:48.447652 12927 heartbeater.cc:503] Master 127.2.32.62:40879 was elected leader, sending a full tablet report...
I20260621 05:21:48.447970 12946 raft_consensus.cc:348] T b4556d755b5a44cdb0b147bf90d1ed7a P 231674478bae4ce5ac9fc3354a412421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c43734595436583e25fd0e453528f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41015 } } peers { permanent_uuid: "231674478bae4ce5ac9fc3354a412421" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36993 } } peers { permanent_uuid: "b331453ac17e4788a59a374767e4869c" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 36713 } }
I20260621 05:21:48.448141 12946 raft_consensus.cc:374] T b4556d755b5a44cdb0b147bf90d1ed7a P 231674478bae4ce5ac9fc3354a412421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:21:48.448205 12946 raft_consensus.cc:729] T b4556d755b5a44cdb0b147bf90d1ed7a P 231674478bae4ce5ac9fc3354a412421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231674478bae4ce5ac9fc3354a412421, State: Initialized, Role: FOLLOWER
I20260621 05:21:48.448315 12946 consensus_queue.cc:260] T b4556d755b5a44cdb0b147bf90d1ed7a P 231674478bae4ce5ac9fc3354a412421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c43734595436583e25fd0e453528f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41015 } } peers { permanent_uuid: "231674478bae4ce5ac9fc3354a412421" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36993 } } peers { permanent_uuid: "b331453ac17e4788a59a374767e4869c" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 36713 } }
I20260621 05:21:48.448601 12946 ts_tablet_manager.cc:1428] T b4556d755b5a44cdb0b147bf90d1ed7a P 231674478bae4ce5ac9fc3354a412421: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260621 05:21:48.448622 12659 heartbeater.cc:503] Master 127.2.32.62:40879 was elected leader, sending a full tablet report...
W20260621 05:21:48.490268 12794 tablet.cc:2369] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:21:48.572721 12660 tablet.cc:2369] T b4556d755b5a44cdb0b147bf90d1ed7a P 231674478bae4ce5ac9fc3354a412421: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:21:48.606695 12950 raft_consensus.cc:482] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:21:48.606830 12950 raft_consensus.cc:504] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c43734595436583e25fd0e453528f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41015 } } peers { permanent_uuid: "231674478bae4ce5ac9fc3354a412421" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36993 } } peers { permanent_uuid: "b331453ac17e4788a59a374767e4869c" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 36713 } }
I20260621 05:21:48.607388 12950 leader_election.cc:288] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 417c43734595436583e25fd0e453528f (127.2.32.3:41015), 231674478bae4ce5ac9fc3354a412421 (127.2.32.1:36993)
I20260621 05:21:48.610664 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:48.607724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:41015 (local address 127.2.32.2:49505)
0621 05:21:48.607811 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:48.607827 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:48.607906 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:48.608090 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:48.608091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:48.608093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:21:48.608177 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:48.608179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.608972 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.608973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:48.609848 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:48.609850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.610402 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.610403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:48.610489 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:48.610492 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:21:48.610552 (+ 60us) client_negotiation.cc:240] Negotiation successful
0621 05:21:48.610595 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":48}
I20260621 05:21:48.610661 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:48.607913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:49505 (local address 127.2.32.3:41015)
0621 05:21:48.607970 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:48.607972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:48.607981 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:21:48.608012 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:48.608012 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:48.608022 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:48.608088 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:21:48.608406 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.608889 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.609941 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.610315 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.610393 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:48.610403 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:48.610558 (+ 155us) server_negotiation.cc:299] Negotiation successful
0621 05:21:48.610592 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 05:21:48.611011 12881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4556d755b5a44cdb0b147bf90d1ed7a" candidate_uuid: "b331453ac17e4788a59a374767e4869c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "417c43734595436583e25fd0e453528f" is_pre_election: true
I20260621 05:21:48.611164 12953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:48.607724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:36993 (local address 127.2.32.2:56751)
0621 05:21:48.608146 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:48.608157 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:21:48.608227 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:21:48.608412 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:21:48.608413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:21:48.608417 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:21:48.608490 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:48.608491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.609150 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.609150 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:48.610209 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:21:48.610214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.610889 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.610890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:21:48.610964 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:48.610968 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:21:48.611030 (+ 62us) client_negotiation.cc:240] Negotiation successful
0621 05:21:48.611071 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":144,"thread_start_us":187,"threads_started":1}
I20260621 05:21:48.611191 12881 raft_consensus.cc:2454] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b331453ac17e4788a59a374767e4869c in term 0.
I20260621 05:21:48.611229 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:48.607914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:56751 (local address 127.2.32.1:36993)
0621 05:21:48.607977 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:48.607980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:48.608242 (+ 262us) server_negotiation.cc:407] Connection header received
0621 05:21:48.608276 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:48.608277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:48.608296 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:48.608349 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:21:48.608607 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.609067 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.610349 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.610790 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.610904 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:48.610920 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:48.611052 (+ 132us) server_negotiation.cc:299] Negotiation successful
0621 05:21:48.611117 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260621 05:21:48.611418 12682 leader_election.cc:302] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417c43734595436583e25fd0e453528f, b331453ac17e4788a59a374767e4869c; no voters:
I20260621 05:21:48.611560 12950 raft_consensus.cc:2781] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:21:48.611609 12950 raft_consensus.cc:482] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:21:48.611657 12950 raft_consensus.cc:3037] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:48.611609 12613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4556d755b5a44cdb0b147bf90d1ed7a" candidate_uuid: "b331453ac17e4788a59a374767e4869c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231674478bae4ce5ac9fc3354a412421" is_pre_election: true
I20260621 05:21:48.611773 12613 raft_consensus.cc:2454] T b4556d755b5a44cdb0b147bf90d1ed7a P 231674478bae4ce5ac9fc3354a412421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b331453ac17e4788a59a374767e4869c in term 0.
I20260621 05:21:48.612524 12950 raft_consensus.cc:504] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c43734595436583e25fd0e453528f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41015 } } peers { permanent_uuid: "231674478bae4ce5ac9fc3354a412421" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36993 } } peers { permanent_uuid: "b331453ac17e4788a59a374767e4869c" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 36713 } }
I20260621 05:21:48.612676 12950 leader_election.cc:288] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [CANDIDATE]: Term 1 election: Requested vote from peers 417c43734595436583e25fd0e453528f (127.2.32.3:41015), 231674478bae4ce5ac9fc3354a412421 (127.2.32.1:36993)
I20260621 05:21:48.612864 12613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4556d755b5a44cdb0b147bf90d1ed7a" candidate_uuid: "b331453ac17e4788a59a374767e4869c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231674478bae4ce5ac9fc3354a412421"
I20260621 05:21:48.612914 12881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4556d755b5a44cdb0b147bf90d1ed7a" candidate_uuid: "b331453ac17e4788a59a374767e4869c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "417c43734595436583e25fd0e453528f"
I20260621 05:21:48.613003 12881 raft_consensus.cc:3037] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:48.612985 12613 raft_consensus.cc:3037] T b4556d755b5a44cdb0b147bf90d1ed7a P 231674478bae4ce5ac9fc3354a412421 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:21:48.613693 12613 raft_consensus.cc:2454] T b4556d755b5a44cdb0b147bf90d1ed7a P 231674478bae4ce5ac9fc3354a412421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b331453ac17e4788a59a374767e4869c in term 1.
I20260621 05:21:48.613770 12881 raft_consensus.cc:2454] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b331453ac17e4788a59a374767e4869c in term 1.
I20260621 05:21:48.613888 12683 leader_election.cc:302] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231674478bae4ce5ac9fc3354a412421, b331453ac17e4788a59a374767e4869c; no voters:
I20260621 05:21:48.614001 12950 raft_consensus.cc:2781] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:21:48.614229 12950 raft_consensus.cc:686] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [term 1 LEADER]: Becoming Leader. State: Replica: b331453ac17e4788a59a374767e4869c, State: Running, Role: LEADER
I20260621 05:21:48.614336 12950 consensus_queue.cc:237] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c43734595436583e25fd0e453528f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41015 } } peers { permanent_uuid: "231674478bae4ce5ac9fc3354a412421" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36993 } } peers { permanent_uuid: "b331453ac17e4788a59a374767e4869c" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 36713 } }
I20260621 05:21:48.615149 12467 catalog_manager.cc:5495] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c reported cstate change: term changed from 0 to 1, leader changed from <none> to b331453ac17e4788a59a374767e4869c (127.2.32.2). New cstate: current_term: 1 leader_uuid: "b331453ac17e4788a59a374767e4869c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417c43734595436583e25fd0e453528f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231674478bae4ce5ac9fc3354a412421" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b331453ac17e4788a59a374767e4869c" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 36713 } health_report { overall_health: HEALTHY } } }
W20260621 05:21:48.649869 12928 tablet.cc:2369] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:21:48.664865 12943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:48.662011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57606 (local address 127.2.32.2:36713)
0621 05:21:48.662122 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0621 05:21:48.662125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:21:48.662139 (+ 14us) server_negotiation.cc:407] Connection header received
0621 05:21:48.662193 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:21:48.662194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:21:48.662222 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:21:48.662323 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:21:48.662449 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.663239 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.664060 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:21:48.664273 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:21:48.664350 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:21:48.664563 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:21:48.664661 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:21:48.664734 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:21:48.664768 (+ 34us) server_negotiation.cc:299] Negotiation successful
0621 05:21:48.664790 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 05:21:48.666946 12881 raft_consensus.cc:1261] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f [term 1 FOLLOWER]: Refusing update from remote peer b331453ac17e4788a59a374767e4869c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:21:48.666958 12613 raft_consensus.cc:1261] T b4556d755b5a44cdb0b147bf90d1ed7a P 231674478bae4ce5ac9fc3354a412421 [term 1 FOLLOWER]: Refusing update from remote peer b331453ac17e4788a59a374767e4869c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:21:48.667420 12954 consensus_queue.cc:1035] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [LEADER]: Connected to new peer: Peer: permanent_uuid: "231674478bae4ce5ac9fc3354a412421" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:21:48.667558 12950 consensus_queue.cc:1035] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [LEADER]: Connected to new peer: Peer: permanent_uuid: "417c43734595436583e25fd0e453528f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:21:48.670840 12959 mvcc.cc:205] Tried to move back new op lower bound from 7299151088296366080 to 7299151088084910080. Current Snapshot: MvccSnapshot[applied={T|T < 7299151088296366080 or (T in {7299151088296366080})}]
I20260621 05:21:48.671442 12958 mvcc.cc:205] Tried to move back new op lower bound from 7299151088296366080 to 7299151088084910080. Current Snapshot: MvccSnapshot[applied={T|T < 7299151088296366080 or (T in {7299151088296366080})}]
I20260621 05:21:48.672253 12467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:21:48.672363 12467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:21:48.673265 12467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9bb75f09ebc4cdfb5ba8675de624706: Sending DeleteTablet for 3 replicas of tablet b4556d755b5a44cdb0b147bf90d1ed7a
I20260621 05:21:48.673641 12861 tablet_service.cc:1514] Processing DeleteTablet for tablet b4556d755b5a44cdb0b147bf90d1ed7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:21:48 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:46242
I20260621 05:21:48.673691 12593 tablet_service.cc:1514] Processing DeleteTablet for tablet b4556d755b5a44cdb0b147bf90d1ed7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:21:48 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:51026
I20260621 05:21:48.673931 12967 tablet_replica.cc:330] T b4556d755b5a44cdb0b147bf90d1ed7a P 231674478bae4ce5ac9fc3354a412421: stopping tablet replica
I20260621 05:21:48.674049 12967 raft_consensus.cc:2229] T b4556d755b5a44cdb0b147bf90d1ed7a P 231674478bae4ce5ac9fc3354a412421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:21:48.674082 12727 tablet_service.cc:1514] Processing DeleteTablet for tablet b4556d755b5a44cdb0b147bf90d1ed7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:21:48 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:57596
I20260621 05:21:48.674176 12967 raft_consensus.cc:2258] T b4556d755b5a44cdb0b147bf90d1ed7a P 231674478bae4ce5ac9fc3354a412421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:21:48.674360 12966 tablet_replica.cc:330] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f: stopping tablet replica
I20260621 05:21:48.674450 12966 raft_consensus.cc:2229] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:21:48.674578 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 12530
I20260621 05:21:48.674602 12967 ts_tablet_manager.cc:1905] T b4556d755b5a44cdb0b147bf90d1ed7a P 231674478bae4ce5ac9fc3354a412421: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:21:48.675397 12968 tablet_replica.cc:330] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c: stopping tablet replica
I20260621 05:21:48.675480 12968 raft_consensus.cc:2229] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:21:48.675558 12968 raft_consensus.cc:2258] T b4556d755b5a44cdb0b147bf90d1ed7a P b331453ac17e4788a59a374767e4869c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:21:48.674535 12966 raft_consensus.cc:2258] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:21:48.676040 12966 ts_tablet_manager.cc:1905] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:21:48.677632 12966 ts_tablet_manager.cc:1918] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:21:48.677708 12966 log.cc:1196] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/b4556d755b5a44cdb0b147bf90d1ed7a
I20260621 05:21:48.677990 12966 ts_tablet_manager.cc:1939] T b4556d755b5a44cdb0b147bf90d1ed7a P 417c43734595436583e25fd0e453528f: Deleting consensus metadata
I20260621 05:21:48.678577 12454 catalog_manager.cc:4841] TS 417c43734595436583e25fd0e453528f (127.2.32.3:41015): tablet b4556d755b5a44cdb0b147bf90d1ed7a (table test-table [id=323912eb64b544fa8544c2d4c68b3f9b]) successfully deleted
W20260621 05:21:48.680877 12455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.32.1:36993 (error 108)
I20260621 05:21:48.681394 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:48.681278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:36993 (local address 127.0.0.1:51040)
0621 05:21:48.681320 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:48.681356 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.1:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260621 05:21:48.681456 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 12664
W20260621 05:21:48.682454 12455 catalog_manager.cc:4584] TS 231674478bae4ce5ac9fc3354a412421 (127.2.32.1:36993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b4556d755b5a44cdb0b147bf90d1ed7a: Network error: Client connection negotiation failed: client connection to 127.2.32.1:36993: connect: Connection refused (error 111)
I20260621 05:21:48.687389 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:21:48.687262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:36713 (local address 127.0.0.1:57616)
0621 05:21:48.687325 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0621 05:21:48.687354 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.2:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 05:21:48.687548 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 12798
I20260621 05:21:48.692286 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 12436
2026-06-21T05:21:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2122 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:21:48 dist-test-slave-b357 krb5kdc[12972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:21:48 dist-test-slave-b357 krb5kdc[12972](info): set up 2 sockets
Jun 21 05:21:48 dist-test-slave-b357 krb5kdc[12972](info): commencing operation
krb5kdc: starting...
W20260621 05:21:49.759433 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:21:49 dist-test-slave-b357 krb5kdc[12972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:21:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:21:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.32.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.32.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.32.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/rangeradmin_127.2.32.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.32.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/rangeradmin_127.2.32.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.32.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.32.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.32.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/rangerlookup_127.2.32.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.32.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/rangerlookup_127.2.32.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.32.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.32.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.32.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/HTTP_127.2.32.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.32.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/HTTP_127.2.32.20@KRBTEST.COM.keytab.
I20260621 05:21:49.934442 2176 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskJjmGzv/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/postgres -l logfile start
2026-06-21 05:21:52.624 UTC [13002] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-21 05:21:52.624 UTC [13002] LOG: listening on IPv4 address "127.2.32.20", port 49733
2026-06-21 05:21:52.628 UTC [13002] LOG: listening on Unix socket "/tmp/.s.PGSQL.49733"
2026-06-21 05:21:52.664 UTC [13005] LOG: database system was shut down at 2026-06-21 05:21:51 UTC
2026-06-21 05:21:52.668 UTC [13002] LOG: database system is ready to accept connections
I20260621 05:21:53.619496 2176 mini_postgres.cc:96] Postgres bound to 49733
2026-06-21 05:21:53.625 UTC [13013] FATAL: database "slave" does not exist
127.2.32.20:49733 - accepting connections
I20260621 05:21:53.626462 2176 mini_ranger.cc:161] Starting Ranger...
I20260621 05:21:53.643466 2176 mini_ranger.cc:84] Created miniranger Postgres user
I20260621 05:21:54.189079 2176 mini_ranger.cc:87] Created ranger Postgres database
I20260621 05:21:54.189201 2176 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ranger-admin
2026-06-21 05:21:54,474 [I] DB FLAVOR :POSTGRES
2026-06-21 05:21:54,474 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 05:21:54,474 [I] Checking connection..
2026-06-21 05:21:54,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-21 05:21:54,733 [I] Checking connection passed.
2026-06-21 05:21:54,733 [I] --------- Verifying version history table ---------
2026-06-21 05:21:54,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 05:21:55,002 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 05:21:55,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 05:21:55,267 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 05:21:55,267 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-21 05:21:55,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-21 05:21:55.532 UTC [13077] WARNING: there is no transaction in progress
2026-06-21 05:21:55,539 [I] create_dbversion_catalog.sql file imported successfully
2026-06-21 05:21:55,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 05:21:55,805 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-21 05:21:55,806 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 05:21:55,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-21 05:21:56,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-21 05:21:56,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-b357.c.gcp-upstream.internal','N') ;"
2026-06-21 05:21:56,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 05:21:56,827 [I] Table x_portal_user does not exist in database ranger
2026-06-21 05:21:56,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 05:21:57,073 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-21 05:21:57,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 05:21:57,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 05:21:57,595 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-21 05:21:57,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-21 05:21:58.595 UTC [13185] WARNING: there is no transaction in progress
2026-06-21 05:21:58.612 UTC [13185] WARNING: there is no transaction in progress
2026-06-21 05:21:58.631 UTC [13185] WARNING: there is no transaction in progress
2026-06-21 05:21:58.647 UTC [13185] WARNING: there is no transaction in progress
2026-06-21 05:21:58.667 UTC [13185] WARNING: there is no transaction in progress
2026-06-21 05:21:58.690 UTC [13185] WARNING: there is no transaction in progress
2026-06-21 05:21:58.752 UTC [13185] WARNING: there is no transaction in progress
2026-06-21 05:21:58.759 UTC [13185] WARNING: there is no transaction in progress
2026-06-21 05:21:58.775 UTC [13185] WARNING: there is no transaction in progress
2026-06-21 05:21:58.786 UTC [13185] WARNING: there is no transaction in progress
2026-06-21 05:21:58.797 UTC [13185] WARNING: there is no transaction in progress
2026-06-21 05:21:58.813 UTC [13185] WARNING: there is no transaction in progress
2026-06-21 05:21:58.824 UTC [13185] WARNING: there is no transaction in progress
2026-06-21 05:21:58.831 UTC [13185] WARNING: there is no transaction in progress
2026-06-21 05:21:58.838 UTC [13185] WARNING: there is no transaction in progress
2026-06-21 05:21:58.846 UTC [13185] WARNING: there is no transaction in progress
2026-06-21 05:21:59,572 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-21 05:21:59,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-21 05:21:59,807 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-21 05:21:59,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 05:22:00,103 [I] Table x_portal_user already exists in database 'ranger'
2026-06-21 05:22:00,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 05:22:00,359 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-21 05:22:00,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 05:22:00,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 05:22:00,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-b357.c.gcp-upstream.internal';"
2026-06-21 05:22:01,131 [I] CORE_DB_SCHEMA import status has been updated
2026-06-21 05:22:01,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-21 05:22:01,393 [I] DB_PATCHES have already been applied
I20260621 05:22:01.398826 2176 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ranger-admin:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskJjmGzv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260621 05:22:01.398902 2176 mini_ranger.cc:193] Using host: 127.2.32.20
I20260621 05:22:01.401978 2176 mini_ranger.cc:227] Ranger admin URL: http://127.2.32.20:56121
Jun 21, 2026 5:22:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 5:22:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 5:22:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ranger-admin/ews/webapp
Jun 21, 2026 5:22:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 21, 2026 5:22:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 21, 2026 5:22:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ranger-admin/ews/webapp].
Jun 21, 2026 5:22:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260621 05:22:02.541900 2176 mini_ranger.cc:160] Time spent starting Ranger: real 8.915s user 0.000s sys 0.010s
Jun 21, 2026 5:22:23 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 21, 2026 5:22:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 21, 2026 5:22:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 21, 2026 5:22:23 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 21, 2026 5:22:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 21, 2026 5:22:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 21, 2026 5:22:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 21, 2026 5:22:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 21, 2026 5:22:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 21, 2026 5:22:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 21, 2026 5:22:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 21, 2026 5:22:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 21, 2026 5:22:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 21, 2026 5:22:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 21, 2026 5:22:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 21, 2026 5:22:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 21, 2026 5:22:23 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 21, 2026 5:22:24 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260621 05:22:24.890225 2176 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.32.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.32.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.32.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/rangerkms_127.2.32.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.32.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/rangerkms_127.2.32.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.32.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.32.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.32.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/HTTP_127.2.32.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.32.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/HTTP_127.2.32.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 21 05:22:24 dist-test-slave-b357 krb5kdc[12972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019344, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260621 05:22:24.978243 2176 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260621 05:22:24.989495 2176 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260621 05:22:25.953162 2176 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260621 05:22:25.953260 2176 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ranger-kms
I20260621 05:22:25.953279 2176 mini_ranger_kms.cc:227] Using postgres at 127.2.32.20:49733
2026-06-21 05:22:26,144 [I] DB FLAVOR :POSTGRES
2026-06-21 05:22:26,144 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 05:22:26,144 [I] Checking connection
2026-06-21 05:22:26,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-21 05:22:26,409 [I] connection success
2026-06-21 05:22:26,409 [I] --------- Verifying Ranger DB tables ---------
2026-06-21 05:22:26,409 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-21 05:22:26,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-21 05:22:26,679 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-21 05:22:26,680 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 05:22:26,680 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-21 05:22:26,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:49733/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-21 05:22:26,946 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260621 05:22:27.496897 2176 mini_ranger_kms.cc:310] Created kms service
I20260621 05:22:27.694629 2176 mini_ranger_kms.cc:326] Created kudu user
I20260621 05:22:27.747644 2176 mini_ranger_kms.cc:343] Created rangerkms user
Jun 21 05:22:27 dist-test-slave-b357 krb5kdc[12972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019344, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
I20260621 05:22:28.235996 2176 mini_ranger_kms.cc:384] Added ranger policy
I20260621 05:22:28.236111 2176 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ranger-kms:/tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260621 05:22:28.236133 2176 mini_ranger_kms.cc:240] Using host: 127.2.32.20
I20260621 05:22:28.239514 2176 mini_ranger_kms.cc:276] Ranger KMS PID: 13366
I20260621 05:22:28.239626 2176 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.32.20:48903
Jun 21, 2026 5:22:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 5:22:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 5:22:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 21, 2026 5:22:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 21, 2026 5:22:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ranger-kms/ews/webapp].
Jun 21, 2026 5:22:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260621 05:22:30.575727 2176 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.598s user 0.003s sys 0.172s
I20260621 05:22:30.575841 2176 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260621 05:22:30.575927 2176 mini_ranger_kms.cc:401] 127.2.32.20:48903/kms/v1/keys
Jun 21 05:22:30 dist-test-slave-b357 krb5kdc[12972](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782019350, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.32.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 21 05:22:31 dist-test-slave-b357 krb5kdc[12972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019350, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.32.20@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
Jun 21 05:22:31 dist-test-slave-b357 krb5kdc[12972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019350, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.32.20@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
Jun 21 05:22:34 dist-test-slave-b357 krb5kdc[12972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:22:34.408648 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:46879
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:39485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:46879
--ranger_config_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.32.20:48903/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:22:34.527657 13416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:22:34.528447 13416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:22:34.528530 13416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:22:34.531430 13416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:22:34.532027 13416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:22:34.532101 13416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:22:34.532128 13416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:22:34.532158 13416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:22:34.534096 13416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:39485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.32.20:48903/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:46879
--ranger_config_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:46879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:22:34.534384 13416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:22:34.534613 13416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:22:34.540807 13421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:22:34.540845 13422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:22:34.540853 13424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:22:34.542477 13416 server_base.cc:1034] running on GCE node
I20260621 05:22:34.543401 13416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:22:34.543944 13416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:22:34.545133 13416 hybrid_clock.cc:648] HybridClock initialized: now 1782019354545069 us; error 83 us; skew 500 ppm
Jun 21 05:22:34 dist-test-slave-b357 krb5kdc[12972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019354, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:22:34.548802 13416 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:22:34.549336 13416 webserver.cc:460] Webserver started at http://127.2.32.62:37883/ using document root <none> and password file <none>
I20260621 05:22:34.549566 13416 fs_manager.cc:359] Metadata directory not provided
I20260621 05:22:34.549607 13416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:22:34.549732 13416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:22:34 dist-test-slave-b357 krb5kdc[12972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019354, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
I20260621 05:22:34.619995 13416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "0a3b8d90ddb54feb87b1bc609de09f0d"
format_stamp: "Formatted at 2026-06-21 05:22:34 on dist-test-slave-b357"
server_key: "8213ecffe58b3d0e83e06a26acfdb43c"
server_key_iv: "94822aed6b8116b51571f647eeaa7973"
server_key_version: "kuduclusterkey@0"
I20260621 05:22:34.620954 13416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "0a3b8d90ddb54feb87b1bc609de09f0d"
format_stamp: "Formatted at 2026-06-21 05:22:34 on dist-test-slave-b357"
server_key: "8213ecffe58b3d0e83e06a26acfdb43c"
server_key_iv: "94822aed6b8116b51571f647eeaa7973"
server_key_version: "kuduclusterkey@0"
I20260621 05:22:34.625852 13416 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260621 05:22:34.640821 13431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:22:34.641031 13416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:22:34.641127 13416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "0a3b8d90ddb54feb87b1bc609de09f0d"
format_stamp: "Formatted at 2026-06-21 05:22:34 on dist-test-slave-b357"
server_key: "8213ecffe58b3d0e83e06a26acfdb43c"
server_key_iv: "94822aed6b8116b51571f647eeaa7973"
server_key_version: "kuduclusterkey@0"
I20260621 05:22:34.641713 13416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:22:34.673621 13416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:22:34.676033 13416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:22:34.676368 13416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:22:34.682080 13416 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:46879
I20260621 05:22:34.682098 13483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:46879 every 8 connection(s)
I20260621 05:22:34.682528 13416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:22:34.683352 13484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:22:34.688884 13484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d: Bootstrap starting.
I20260621 05:22:34.688900 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 13416
I20260621 05:22:34.689100 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:22:34.689419 2176 external_mini_cluster.cc:1428] Setting key a839c6d5cfa11724a9ca400c86d79e16
I20260621 05:22:34.689898 13484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d: Neither blocks nor log segments found. Creating new log.
I20260621 05:22:34.690282 13484 log.cc:824] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d: Log is configured to *not* fsync() on all Append() calls
I20260621 05:22:34.691222 13484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d: No bootstrap required, opened a new log
I20260621 05:22:34.693082 13484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a3b8d90ddb54feb87b1bc609de09f0d" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46879 } }
I20260621 05:22:34.693244 13484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:22:34.693284 13484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a3b8d90ddb54feb87b1bc609de09f0d, State: Initialized, Role: FOLLOWER
I20260621 05:22:34.693449 13484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a3b8d90ddb54feb87b1bc609de09f0d" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46879 } }
I20260621 05:22:34.693548 13484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:22:34.693604 13484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:22:34.693653 13484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:22:34.694321 13484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a3b8d90ddb54feb87b1bc609de09f0d" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46879 } }
I20260621 05:22:34.694422 13484 leader_election.cc:302] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a3b8d90ddb54feb87b1bc609de09f0d; no voters:
I20260621 05:22:34.694603 13484 leader_election.cc:288] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:22:34.694653 13489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:22:34.694869 13489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d [term 1 LEADER]: Becoming Leader. State: Replica: 0a3b8d90ddb54feb87b1bc609de09f0d, State: Running, Role: LEADER
I20260621 05:22:34.694896 13484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:22:34.695639 13489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a3b8d90ddb54feb87b1bc609de09f0d" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46879 } }
I20260621 05:22:34.696462 13490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a3b8d90ddb54feb87b1bc609de09f0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a3b8d90ddb54feb87b1bc609de09f0d" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46879 } } }
I20260621 05:22:34.696530 13490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d [sys.catalog]: This master's current role is: LEADER
I20260621 05:22:34.696601 13484 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Jun 21 05:22:34 dist-test-slave-b357 krb5kdc[12972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:22:34.701846 13503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:22:34.702443 13491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a3b8d90ddb54feb87b1bc609de09f0d. Latest consensus state: current_term: 1 leader_uuid: "0a3b8d90ddb54feb87b1bc609de09f0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a3b8d90ddb54feb87b1bc609de09f0d" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46879 } } }
I20260621 05:22:34.702625 13491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d [sys.catalog]: This master's current role is: LEADER
I20260621 05:22:34.703192 13503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:22:34.707118 13487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:22:34.690050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33656 (local address 127.2.32.62:46879)
0621 05:22:34.690270 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 05:22:34.690279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:22:34.690311 (+ 32us) server_negotiation.cc:407] Connection header received
0621 05:22:34.691031 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:22:34.691038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:22:34.691089 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:22:34.691317 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:22:34.692078 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:34.693321 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:34.694181 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:34.694729 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:34.695167 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:22:34.699602 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:22:34.699649 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:22:34.699650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:22:34.700259 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:22:34.704543 (+ 4284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:22:34.704964 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:22:34.704966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:22:34.704967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:22:34.705067 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:22:34.705328 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:22:34.705329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:22:34.705329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:22:34.706054 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:22:34.706654 (+ 600us) server_negotiation.cc:1035] Waiting for connection context
0621 05:22:34.706729 (+ 75us) server_negotiation.cc:299] Negotiation successful
0621 05:22:34.706884 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":79,"threads_started":1}
I20260621 05:22:34.711773 13503 catalog_manager.cc:1347] Generated new cluster ID: 0b1032c9f22a4604801b6b8aefc25080
I20260621 05:22:34.711833 13503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:22:34.719645 13503 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:22:34.720566 13503 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:22:34.727420 13503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d: Generated new TSK 0
I20260621 05:22:34.728041 13503 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
I20260621 05:22:35.556434 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.32.20:48903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46879
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:39485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:22:35.663416 13529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:22:35.663664 13529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:22:35.663708 13529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:22:35.665321 13529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:22:35.665382 13529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:22:35.665433 13529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:22:35.666970 13529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:39485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.32.20:48903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:22:35.667259 13529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:22:35.667500 13529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:22:35.675323 13537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:22:35.675061 13534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:22:35.675410 13535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:22:35.676023 13529 server_base.cc:1034] running on GCE node
I20260621 05:22:35.676353 13529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:22:35.676693 13529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:22:35.677935 13529 hybrid_clock.cc:648] HybridClock initialized: now 1782019355677920 us; error 33 us; skew 500 ppm
Jun 21 05:22:35 dist-test-slave-b357 krb5kdc[12972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019355, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:22:35.682771 13529 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:22:35.683437 13529 webserver.cc:460] Webserver started at http://127.2.32.1:38085/ using document root <none> and password file <none>
I20260621 05:22:35.683741 13529 fs_manager.cc:359] Metadata directory not provided
I20260621 05:22:35.683882 13529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:22:35.684139 13529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:22:35 dist-test-slave-b357 krb5kdc[12972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019355, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
I20260621 05:22:35.701225 13529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "bb30c8b777354f6abd0cba489d494d39"
format_stamp: "Formatted at 2026-06-21 05:22:35 on dist-test-slave-b357"
server_key: "f1caf95ad9cdc2d32b92af3bab02f19e"
server_key_iv: "e6bed34ad7e2d9df5a41e4777e4135cd"
server_key_version: "kuduclusterkey@0"
I20260621 05:22:35.701819 13529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "bb30c8b777354f6abd0cba489d494d39"
format_stamp: "Formatted at 2026-06-21 05:22:35 on dist-test-slave-b357"
server_key: "f1caf95ad9cdc2d32b92af3bab02f19e"
server_key_iv: "e6bed34ad7e2d9df5a41e4777e4135cd"
server_key_version: "kuduclusterkey@0"
I20260621 05:22:35.708792 13529 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.000s
I20260621 05:22:35.726754 13544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:22:35.726936 13529 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:22:35.727025 13529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "bb30c8b777354f6abd0cba489d494d39"
format_stamp: "Formatted at 2026-06-21 05:22:35 on dist-test-slave-b357"
server_key: "f1caf95ad9cdc2d32b92af3bab02f19e"
server_key_iv: "e6bed34ad7e2d9df5a41e4777e4135cd"
server_key_version: "kuduclusterkey@0"
I20260621 05:22:35.727150 13529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:22:35.742786 13529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:22:35.745841 13529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:22:35.746212 13529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:22:35.746570 13529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:22:35.747205 13529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:22:35.747370 13529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:22:35.747475 13529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:22:35.747555 13529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:22:35.755822 13529 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:38077
I20260621 05:22:35.756196 13529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:22:35.759517 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 13529
I20260621 05:22:35.759641 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:22:35.759845 2176 external_mini_cluster.cc:1428] Setting key dbe0d370f3e7e8f901b885118128dbb4
I20260621 05:22:35.757021 13659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:38077 every 8 connection(s)
WARNING: no policy specified for kudu/127.2.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.2@KRBTEST.COM" created.
Jun 21 05:22:35 dist-test-slave-b357 krb5kdc[12972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019355, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:22:35.777784 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:22:35.757608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:59925 (local address 127.2.32.62:46879)
0621 05:22:35.764390 (+ 6782us) server_negotiation.cc:206] Beginning negotiation
0621 05:22:35.764393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:22:35.764402 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:22:35.764428 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:22:35.764429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:22:35.764442 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:22:35.764503 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:22:35.765798 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:35.766676 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:35.768258 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:35.768586 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:35.768669 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:22:35.773556 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:22:35.773558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:22:35.773558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:22:35.773570 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:22:35.775600 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:22:35.776079 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:22:35.776080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:22:35.776080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:22:35.776148 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:22:35.776642 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:22:35.776645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:22:35.776646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:22:35.777024 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:22:35.777114 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 05:22:35.777514 (+ 400us) server_negotiation.cc:299] Negotiation successful
0621 05:22:35.777620 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6757,"thread_start_us":71,"threads_started":1}
I20260621 05:22:35.777859 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:22:35.757596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:46879 (local address 127.2.32.1:59925)
0621 05:22:35.763395 (+ 5799us) negotiation.cc:107] Waiting for socket to connect
0621 05:22:35.763439 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 05:22:35.764196 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:22:35.764650 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:22:35.764655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:22:35.765372 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:22:35.765676 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:22:35.765682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:22:35.766785 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:22:35.766786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:22:35.768156 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:22:35.768158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:22:35.768740 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:22:35.768741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:22:35.768891 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:22:35.769572 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:22:35.769583 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:22:35.773330 (+ 3747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:22:35.775711 (+ 2381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:22:35.775718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:22:35.775721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:22:35.775983 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:22:35.776227 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:22:35.776229 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:22:35.776229 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:22:35.776381 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:22:35.777120 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:22:35.777126 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:22:35.777319 (+ 193us) client_negotiation.cc:769] Sending connection context
0621 05:22:35.777445 (+ 126us) client_negotiation.cc:240] Negotiation successful
0621 05:22:35.777618 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5750,"thread_start_us":75,"threads_started":1}
I20260621 05:22:35.778537 13660 heartbeater.cc:347] Connected to a master server at 127.2.32.62:46879
I20260621 05:22:35.778648 13660 heartbeater.cc:464] Registering TS with master...
I20260621 05:22:35.778849 13660 heartbeater.cc:511] Master 127.2.32.62:46879 requested a full tablet report, sending...
I20260621 05:22:35.779466 13448 ts_manager.cc:194] Registered new tserver with Master: bb30c8b777354f6abd0cba489d494d39 (127.2.32.1:38077)
I20260621 05:22:35.780417 13448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:59925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
I20260621 05:22:35.793776 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.32.20:48903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46879
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:39485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:22:35.911422 13666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:22:35.911872 13666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:22:35.912052 13666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:22:35.913956 13666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:22:35.914045 13666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:22:35.914103 13666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:22:35.916245 13666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:39485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.32.20:48903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:22:35.916594 13666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:22:35.916872 13666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:22:35.922680 13673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:22:35.922780 13674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:22:35.922680 13676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:22:35.924022 13666 server_base.cc:1034] running on GCE node
I20260621 05:22:35.924232 13666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:22:35.924479 13666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:22:35.925657 13666 hybrid_clock.cc:648] HybridClock initialized: now 1782019355925609 us; error 66 us; skew 500 ppm
Jun 21 05:22:35 dist-test-slave-b357 krb5kdc[12972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019355, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:22:35.929137 13666 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:22:35.929673 13666 webserver.cc:460] Webserver started at http://127.2.32.2:33083/ using document root <none> and password file <none>
I20260621 05:22:35.929976 13666 fs_manager.cc:359] Metadata directory not provided
I20260621 05:22:35.930069 13666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:22:35.930208 13666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:22:35 dist-test-slave-b357 krb5kdc[12972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019355, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
I20260621 05:22:35.944348 13666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "21ce7b6f45294fab8462dbf5bca42d94"
format_stamp: "Formatted at 2026-06-21 05:22:35 on dist-test-slave-b357"
server_key: "60984e4bb711dbc9b710dbeda0d426b3"
server_key_iv: "fa152f725f2b33ddf86c91578810cfc2"
server_key_version: "kuduclusterkey@0"
I20260621 05:22:35.944850 13666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "21ce7b6f45294fab8462dbf5bca42d94"
format_stamp: "Formatted at 2026-06-21 05:22:35 on dist-test-slave-b357"
server_key: "60984e4bb711dbc9b710dbeda0d426b3"
server_key_iv: "fa152f725f2b33ddf86c91578810cfc2"
server_key_version: "kuduclusterkey@0"
I20260621 05:22:35.946946 13666 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
Jun 21 05:22:35 dist-test-slave-b357 krb5kdc[12972](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782019355, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:22:35.967049 13683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:22:35.967314 13666 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260621 05:22:35.967437 13666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "21ce7b6f45294fab8462dbf5bca42d94"
format_stamp: "Formatted at 2026-06-21 05:22:35 on dist-test-slave-b357"
server_key: "60984e4bb711dbc9b710dbeda0d426b3"
server_key_iv: "fa152f725f2b33ddf86c91578810cfc2"
server_key_version: "kuduclusterkey@0"
I20260621 05:22:35.967592 13666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:22:36.029656 13666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:22:36.033318 13666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:22:36.033614 13666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:22:36.034045 13666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:22:36.034614 13666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:22:36.034683 13666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:22:36.034751 13666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:22:36.034781 13666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:22:36.045591 13666 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:45719
I20260621 05:22:36.045672 13797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:45719 every 8 connection(s)
I20260621 05:22:36.046293 13666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:22:36.047868 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 13666
I20260621 05:22:36.048007 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:22:36.048245 2176 external_mini_cluster.cc:1428] Setting key 4ab264619d3bf1e39d3af1c78afe0c99
WARNING: no policy specified for kudu/127.2.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.3@KRBTEST.COM" created.
Jun 21 05:22:36 dist-test-slave-b357 krb5kdc[12972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019355, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:22:36.067476 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:22:36.048174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:54841 (local address 127.2.32.62:46879)
0621 05:22:36.048225 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0621 05:22:36.048227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:22:36.048412 (+ 185us) server_negotiation.cc:407] Connection header received
0621 05:22:36.049140 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:22:36.049142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:22:36.049153 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:22:36.049186 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:22:36.050046 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.051086 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.055278 (+ 4192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.055627 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.055685 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:22:36.063916 (+ 8231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:22:36.063918 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:22:36.063918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:22:36.063928 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:22:36.065744 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:22:36.066270 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:22:36.066272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:22:36.066272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:22:36.066331 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:22:36.066630 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:22:36.066631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:22:36.066632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:22:36.066780 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:22:36.066823 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0621 05:22:36.067258 (+ 435us) server_negotiation.cc:299] Negotiation successful
0621 05:22:36.067355 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:22:36.068011 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:22:36.047871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:46879 (local address 127.2.32.2:54841)
0621 05:22:36.048289 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0621 05:22:36.048334 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 05:22:36.049004 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:22:36.049364 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:22:36.049368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:22:36.049655 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:22:36.049931 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:22:36.049936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.051244 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.051245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:22:36.054498 (+ 3253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:22:36.054500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.055885 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.055887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:22:36.056101 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:22:36.057521 (+ 1420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:22:36.057536 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:22:36.063774 (+ 6238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:22:36.065844 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:22:36.065853 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:22:36.065856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:22:36.066176 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:22:36.066405 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:22:36.066406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:22:36.066406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:22:36.066541 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:22:36.066870 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:22:36.066876 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:22:36.067064 (+ 188us) client_negotiation.cc:769] Sending connection context
0621 05:22:36.067583 (+ 519us) client_negotiation.cc:240] Negotiation successful
0621 05:22:36.067759 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":106,"threads_started":1}
I20260621 05:22:36.068588 13798 heartbeater.cc:347] Connected to a master server at 127.2.32.62:46879
I20260621 05:22:36.068689 13798 heartbeater.cc:464] Registering TS with master...
I20260621 05:22:36.068890 13798 heartbeater.cc:511] Master 127.2.32.62:46879 requested a full tablet report, sending...
I20260621 05:22:36.069361 13448 ts_manager.cc:194] Registered new tserver with Master: 21ce7b6f45294fab8462dbf5bca42d94 (127.2.32.2:45719)
I20260621 05:22:36.069888 13448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:54841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
I20260621 05:22:36.087920 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.32.20:48903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46879
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:39485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:22:36.220217 13803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:22:36.220521 13803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:22:36.220561 13803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:22:36.222659 13803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:22:36.222726 13803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:22:36.222769 13803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:22:36.225091 13803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:39485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.32.20:48903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:22:36.225354 13803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:22:36.225631 13803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:22:36.233246 13811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:22:36.233455 13808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:22:36.233544 13809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:22:36.234732 13803 server_base.cc:1034] running on GCE node
I20260621 05:22:36.234942 13803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:22:36.235227 13803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:22:36.236418 13803 hybrid_clock.cc:648] HybridClock initialized: now 1782019356236394 us; error 35 us; skew 500 ppm
Jun 21 05:22:36 dist-test-slave-b357 krb5kdc[12972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019356, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:22:36.240061 13803 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:22:36.240870 13803 webserver.cc:460] Webserver started at http://127.2.32.3:38769/ using document root <none> and password file <none>
I20260621 05:22:36.241147 13803 fs_manager.cc:359] Metadata directory not provided
I20260621 05:22:36.241219 13803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:22:36.241356 13803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:22:36 dist-test-slave-b357 krb5kdc[12972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019356, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
I20260621 05:22:36.261305 13803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "7a6a0b7372f349fd908ccb851cb72b61"
format_stamp: "Formatted at 2026-06-21 05:22:36 on dist-test-slave-b357"
server_key: "dfc999fcbbc6a716474f761e59e8a89d"
server_key_iv: "f13cb2b8fbb8f1be4f777fe715bd4570"
server_key_version: "kuduclusterkey@0"
I20260621 05:22:36.261765 13803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "7a6a0b7372f349fd908ccb851cb72b61"
format_stamp: "Formatted at 2026-06-21 05:22:36 on dist-test-slave-b357"
server_key: "dfc999fcbbc6a716474f761e59e8a89d"
server_key_iv: "f13cb2b8fbb8f1be4f777fe715bd4570"
server_key_version: "kuduclusterkey@0"
I20260621 05:22:36.267549 13803 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.002s
I20260621 05:22:36.283047 13818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:22:36.283362 13803 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:22:36.283525 13803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "7a6a0b7372f349fd908ccb851cb72b61"
format_stamp: "Formatted at 2026-06-21 05:22:36 on dist-test-slave-b357"
server_key: "dfc999fcbbc6a716474f761e59e8a89d"
server_key_iv: "f13cb2b8fbb8f1be4f777fe715bd4570"
server_key_version: "kuduclusterkey@0"
I20260621 05:22:36.283727 13803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:22:36.311760 13803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:22:36.314266 13803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:22:36.314420 13803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:22:36.314661 13803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:22:36.315006 13803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:22:36.315053 13803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:22:36.315102 13803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:22:36.315146 13803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:22:36.322036 13803 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:37205
I20260621 05:22:36.322520 13803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:22:36.324923 13931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:37205 every 8 connection(s)
I20260621 05:22:36.325366 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 13803
I20260621 05:22:36.325608 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:22:36.325879 2176 external_mini_cluster.cc:1428] Setting key f5e3b3d691ec8d3c6d655c3473c282b7
Jun 21 05:22:36 dist-test-slave-b357 krb5kdc[12972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019356, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:22:36.338234 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:22:36.325893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:53949 (local address 127.2.32.62:46879)
0621 05:22:36.325937 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:22:36.325939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:22:36.327433 (+ 1494us) server_negotiation.cc:407] Connection header received
0621 05:22:36.328485 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:22:36.328487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:22:36.328505 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:22:36.328541 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:22:36.329681 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.330259 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.331509 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.331718 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.331833 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:22:36.334093 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:22:36.334095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:22:36.334095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:22:36.334108 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:22:36.336002 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:22:36.336547 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:22:36.336549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:22:36.336549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:22:36.336613 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:22:36.336950 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:22:36.336951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:22:36.336951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:22:36.337574 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:22:36.337624 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0621 05:22:36.338032 (+ 408us) server_negotiation.cc:299] Negotiation successful
0621 05:22:36.338131 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:22:36.339159 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:22:36.325594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:46879 (local address 127.2.32.3:53949)
0621 05:22:36.327288 (+ 1694us) negotiation.cc:107] Waiting for socket to connect
0621 05:22:36.327335 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 05:22:36.328260 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:22:36.328814 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:22:36.328820 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:22:36.329135 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:22:36.329439 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:22:36.329444 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.330376 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.330378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:22:36.331403 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:22:36.331406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.331843 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.331844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:22:36.331994 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:22:36.332606 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:22:36.332630 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:22:36.333976 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:22:36.336125 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:22:36.336131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:22:36.336133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:22:36.336444 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:22:36.336697 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:22:36.336698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:22:36.336698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:22:36.336829 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:22:36.337678 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:22:36.337684 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:22:36.337861 (+ 177us) client_negotiation.cc:769] Sending connection context
0621 05:22:36.338713 (+ 852us) client_negotiation.cc:240] Negotiation successful
0621 05:22:36.338883 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1649,"thread_start_us":90,"threads_started":1}
I20260621 05:22:36.339840 13932 heartbeater.cc:347] Connected to a master server at 127.2.32.62:46879
I20260621 05:22:36.339983 13932 heartbeater.cc:464] Registering TS with master...
I20260621 05:22:36.340230 13932 heartbeater.cc:511] Master 127.2.32.62:46879 requested a full tablet report, sending...
I20260621 05:22:36.340633 13448 ts_manager.cc:194] Registered new tserver with Master: 7a6a0b7372f349fd908ccb851cb72b61 (127.2.32.3:37205)
I20260621 05:22:36.341284 13448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:53949
I20260621 05:22:36.348462 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:22:36.359869 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:22:36.350599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33662 (local address 127.2.32.62:46879)
0621 05:22:36.350650 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0621 05:22:36.350652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:22:36.351070 (+ 418us) server_negotiation.cc:407] Connection header received
0621 05:22:36.351115 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:22:36.351116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:22:36.351159 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:22:36.351259 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:22:36.352068 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.352934 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.354105 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.354427 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.354535 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:22:36.356306 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:22:36.356308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:22:36.356308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:22:36.356317 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:22:36.358368 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:22:36.358839 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:22:36.358840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:22:36.358840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:22:36.358922 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:22:36.359219 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:22:36.359220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:22:36.359220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:22:36.359369 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:22:36.359486 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 05:22:36.359682 (+ 196us) server_negotiation.cc:299] Negotiation successful
0621 05:22:36.359770 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:22:36.361974 13448 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:22:36.364462 13448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:22:36.379681 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:22:36.373480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:38077 (local address 127.0.0.1:53634)
0621 05:22:36.375279 (+ 1799us) negotiation.cc:107] Waiting for socket to connect
0621 05:22:36.375317 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 05:22:36.375384 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:22:36.375657 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:22:36.375659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:22:36.375661 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:22:36.375734 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:22:36.375736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.377012 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.377013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:22:36.378256 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:22:36.378259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.379353 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.379354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:22:36.379444 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:22:36.379458 (+ 14us) client_negotiation.cc:769] Sending connection context
0621 05:22:36.379529 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 05:22:36.379592 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1765,"thread_start_us":83,"threads_started":1}
I20260621 05:22:36.379920 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:22:36.373999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53634 (local address 127.2.32.1:38077)
0621 05:22:36.375369 (+ 1370us) server_negotiation.cc:206] Beginning negotiation
0621 05:22:36.375373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:22:36.375386 (+ 13us) server_negotiation.cc:407] Connection header received
0621 05:22:36.375495 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:22:36.375499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:22:36.375515 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:22:36.375565 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:22:36.375824 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.376895 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.378375 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.379171 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.379227 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:22:36.379284 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 05:22:36.379810 (+ 526us) server_negotiation.cc:299] Negotiation successful
0621 05:22:36.379851 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1342,"thread_start_us":74,"threads_started":1}
I20260621 05:22:36.380662 13594 tablet_service.cc:1467] Processing CreateTablet for tablet 8889beb66f374264b1453fe0ebd43066 (DEFAULT_TABLE table=test-table [id=81c300af17e3477388016cff1afa12b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:22:36.381047 13594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8889beb66f374264b1453fe0ebd43066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:22:36.384809 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:22:36.373831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53110 (local address 127.2.32.3:37205)
0621 05:22:36.375298 (+ 1467us) server_negotiation.cc:206] Beginning negotiation
0621 05:22:36.375304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:22:36.375884 (+ 580us) server_negotiation.cc:407] Connection header received
0621 05:22:36.375992 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:22:36.375994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:22:36.376009 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:22:36.376058 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:22:36.376581 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.377678 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.380993 (+ 3315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.382125 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.382174 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:22:36.382244 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:22:36.384669 (+ 2425us) server_negotiation.cc:299] Negotiation successful
0621 05:22:36.384725 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1428,"thread_start_us":92,"threads_started":1}
I20260621 05:22:36.385043 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:22:36.373493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:37205 (local address 127.0.0.1:53110)
0621 05:22:36.375832 (+ 2339us) negotiation.cc:107] Waiting for socket to connect
0621 05:22:36.375841 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:22:36.375907 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:22:36.376450 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:22:36.376451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:22:36.376453 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:22:36.376501 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:22:36.376503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.377776 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.377777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:22:36.379257 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:22:36.379259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.382306 (+ 3047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.382307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:22:36.382380 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:22:36.382383 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:22:36.384930 (+ 2547us) client_negotiation.cc:240] Negotiation successful
0621 05:22:36.384973 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2324,"thread_start_us":107,"threads_started":1}
I20260621 05:22:36.385766 13866 tablet_service.cc:1467] Processing CreateTablet for tablet 8889beb66f374264b1453fe0ebd43066 (DEFAULT_TABLE table=test-table [id=81c300af17e3477388016cff1afa12b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:22:36.386194 13866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8889beb66f374264b1453fe0ebd43066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:22:36.386610 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:22:36.373743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:45719 (local address 127.0.0.1:51988)
0621 05:22:36.379362 (+ 5619us) negotiation.cc:107] Waiting for socket to connect
0621 05:22:36.379371 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:22:36.379416 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:22:36.379662 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:22:36.379663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:22:36.379665 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:22:36.379698 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:22:36.379699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.383248 (+ 3549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.383249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:22:36.384357 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:22:36.384359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.386384 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.386385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:22:36.386444 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:22:36.386448 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:22:36.386502 (+ 54us) client_negotiation.cc:240] Negotiation successful
0621 05:22:36.386542 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5603,"thread_start_us":64,"threads_started":1}
I20260621 05:22:36.386897 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:22:36.373957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.2.32.2:45719)
0621 05:22:36.375460 (+ 1503us) server_negotiation.cc:206] Beginning negotiation
0621 05:22:36.375464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:22:36.379470 (+ 4006us) server_negotiation.cc:407] Connection header received
0621 05:22:36.379510 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:22:36.379512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:22:36.379527 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:22:36.379592 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:22:36.380351 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.383152 (+ 2801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.385468 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.386199 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.386272 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:22:36.386348 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:22:36.386778 (+ 430us) server_negotiation.cc:299] Negotiation successful
0621 05:22:36.386820 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1472,"thread_start_us":102,"threads_started":1}
I20260621 05:22:36.387449 13730 tablet_service.cc:1467] Processing CreateTablet for tablet 8889beb66f374264b1453fe0ebd43066 (DEFAULT_TABLE table=test-table [id=81c300af17e3477388016cff1afa12b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:22:36.387764 13730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8889beb66f374264b1453fe0ebd43066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:22:36.390807 13951 tablet_bootstrap.cc:492] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61: Bootstrap starting.
I20260621 05:22:36.391719 13952 tablet_bootstrap.cc:492] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94: Bootstrap starting.
I20260621 05:22:36.391969 13951 tablet_bootstrap.cc:654] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61: Neither blocks nor log segments found. Creating new log.
I20260621 05:22:36.392351 13951 log.cc:824] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61: Log is configured to *not* fsync() on all Append() calls
I20260621 05:22:36.392671 13952 tablet_bootstrap.cc:654] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94: Neither blocks nor log segments found. Creating new log.
I20260621 05:22:36.392987 13952 log.cc:824] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94: Log is configured to *not* fsync() on all Append() calls
I20260621 05:22:36.393144 13951 tablet_bootstrap.cc:492] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61: No bootstrap required, opened a new log
I20260621 05:22:36.393198 13951 ts_tablet_manager.cc:1397] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:22:36.393637 13952 tablet_bootstrap.cc:492] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94: No bootstrap required, opened a new log
I20260621 05:22:36.393705 13952 ts_tablet_manager.cc:1397] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:22:36.395309 13951 raft_consensus.cc:348] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6a0b7372f349fd908ccb851cb72b61" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37205 } } peers { permanent_uuid: "21ce7b6f45294fab8462dbf5bca42d94" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45719 } } peers { permanent_uuid: "bb30c8b777354f6abd0cba489d494d39" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 38077 } }
I20260621 05:22:36.395467 13951 raft_consensus.cc:374] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:22:36.395502 13951 raft_consensus.cc:729] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6a0b7372f349fd908ccb851cb72b61, State: Initialized, Role: FOLLOWER
I20260621 05:22:36.395608 13951 consensus_queue.cc:260] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6a0b7372f349fd908ccb851cb72b61" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37205 } } peers { permanent_uuid: "21ce7b6f45294fab8462dbf5bca42d94" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45719 } } peers { permanent_uuid: "bb30c8b777354f6abd0cba489d494d39" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 38077 } }
I20260621 05:22:36.395560 13952 raft_consensus.cc:348] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6a0b7372f349fd908ccb851cb72b61" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37205 } } peers { permanent_uuid: "21ce7b6f45294fab8462dbf5bca42d94" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45719 } } peers { permanent_uuid: "bb30c8b777354f6abd0cba489d494d39" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 38077 } }
I20260621 05:22:36.395709 13952 raft_consensus.cc:374] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:22:36.395749 13952 raft_consensus.cc:729] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21ce7b6f45294fab8462dbf5bca42d94, State: Initialized, Role: FOLLOWER
I20260621 05:22:36.395841 13951 ts_tablet_manager.cc:1428] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260621 05:22:36.395845 13952 consensus_queue.cc:260] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6a0b7372f349fd908ccb851cb72b61" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37205 } } peers { permanent_uuid: "21ce7b6f45294fab8462dbf5bca42d94" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45719 } } peers { permanent_uuid: "bb30c8b777354f6abd0cba489d494d39" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 38077 } }
I20260621 05:22:36.396065 13952 ts_tablet_manager.cc:1428] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:22:36.396204 13798 heartbeater.cc:503] Master 127.2.32.62:46879 was elected leader, sending a full tablet report...
I20260621 05:22:36.396608 13932 heartbeater.cc:503] Master 127.2.32.62:46879 was elected leader, sending a full tablet report...
I20260621 05:22:36.399329 13950 tablet_bootstrap.cc:492] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39: Bootstrap starting.
I20260621 05:22:36.400067 13950 tablet_bootstrap.cc:654] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39: Neither blocks nor log segments found. Creating new log.
I20260621 05:22:36.400360 13950 log.cc:824] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39: Log is configured to *not* fsync() on all Append() calls
I20260621 05:22:36.401084 13950 tablet_bootstrap.cc:492] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39: No bootstrap required, opened a new log
I20260621 05:22:36.401142 13950 ts_tablet_manager.cc:1397] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:22:36.404100 13950 raft_consensus.cc:348] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6a0b7372f349fd908ccb851cb72b61" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37205 } } peers { permanent_uuid: "21ce7b6f45294fab8462dbf5bca42d94" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45719 } } peers { permanent_uuid: "bb30c8b777354f6abd0cba489d494d39" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 38077 } }
I20260621 05:22:36.404282 13950 raft_consensus.cc:374] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:22:36.404320 13950 raft_consensus.cc:729] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb30c8b777354f6abd0cba489d494d39, State: Initialized, Role: FOLLOWER
I20260621 05:22:36.404773 13950 consensus_queue.cc:260] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6a0b7372f349fd908ccb851cb72b61" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37205 } } peers { permanent_uuid: "21ce7b6f45294fab8462dbf5bca42d94" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45719 } } peers { permanent_uuid: "bb30c8b777354f6abd0cba489d494d39" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 38077 } }
I20260621 05:22:36.405057 13950 ts_tablet_manager.cc:1428] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260621 05:22:36.405184 13660 heartbeater.cc:503] Master 127.2.32.62:46879 was elected leader, sending a full tablet report...
Jun 21 05:22:36 dist-test-slave-b357 krb5kdc[12972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019355, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
I20260621 05:22:36.479997 13958 raft_consensus.cc:482] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:22:36.480149 13958 raft_consensus.cc:504] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6a0b7372f349fd908ccb851cb72b61" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37205 } } peers { permanent_uuid: "21ce7b6f45294fab8462dbf5bca42d94" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45719 } } peers { permanent_uuid: "bb30c8b777354f6abd0cba489d494d39" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 38077 } }
I20260621 05:22:36.480515 13958 leader_election.cc:288] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a6a0b7372f349fd908ccb851cb72b61 (127.2.32.3:37205), 21ce7b6f45294fab8462dbf5bca42d94 (127.2.32.2:45719)
I20260621 05:22:36.492442 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:22:36.481154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:45719 (local address 127.2.32.1:49547)
0621 05:22:36.488930 (+ 7776us) negotiation.cc:107] Waiting for socket to connect
0621 05:22:36.488942 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:22:36.489034 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:22:36.489233 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:22:36.489234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:22:36.489236 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:22:36.489314 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:22:36.489315 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.490255 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.490256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:22:36.491438 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:22:36.491441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.492193 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.492194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:22:36.492260 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:22:36.492264 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:22:36.492313 (+ 49us) client_negotiation.cc:240] Negotiation successful
0621 05:22:36.492351 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7742,"thread_start_us":66,"threads_started":1}
I20260621 05:22:36.492501 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:22:36.481382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:49547 (local address 127.2.32.2:45719)
0621 05:22:36.481454 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0621 05:22:36.481456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:22:36.489005 (+ 7549us) server_negotiation.cc:407] Connection header received
0621 05:22:36.489129 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:22:36.489130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:22:36.489143 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:22:36.489170 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:22:36.489375 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.490116 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.491533 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.492042 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.492123 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:22:36.492135 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0621 05:22:36.492348 (+ 213us) server_negotiation.cc:299] Negotiation successful
0621 05:22:36.492395 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 05:22:36.495705 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:22:36.481343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:49609 (local address 127.2.32.3:37205)
0621 05:22:36.481424 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0621 05:22:36.481426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:22:36.484160 (+ 2734us) server_negotiation.cc:407] Connection header received
0621 05:22:36.484200 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:22:36.484201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:22:36.484213 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:22:36.484298 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:22:36.493225 (+ 8927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.493720 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.494678 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.495268 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.495347 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:22:36.495358 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0621 05:22:36.495563 (+ 205us) server_negotiation.cc:299] Negotiation successful
0621 05:22:36.495608 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 05:22:36.495985 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:22:36.480978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:37205 (local address 127.2.32.1:49609)
0621 05:22:36.484051 (+ 3073us) negotiation.cc:107] Waiting for socket to connect
0621 05:22:36.484063 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:22:36.484132 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:22:36.492696 (+ 8564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:22:36.492698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:22:36.492706 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:22:36.492760 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:22:36.492762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.493817 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.493818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:22:36.494569 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:22:36.494571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.495395 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.495396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:22:36.495467 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:22:36.495472 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:22:36.495837 (+ 365us) client_negotiation.cc:240] Negotiation successful
0621 05:22:36.495883 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3052,"thread_start_us":87,"threads_started":1}
I20260621 05:22:36.496208 13751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8889beb66f374264b1453fe0ebd43066" candidate_uuid: "bb30c8b777354f6abd0cba489d494d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ce7b6f45294fab8462dbf5bca42d94" is_pre_election: true
I20260621 05:22:36.496271 13886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8889beb66f374264b1453fe0ebd43066" candidate_uuid: "bb30c8b777354f6abd0cba489d494d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6a0b7372f349fd908ccb851cb72b61" is_pre_election: true
I20260621 05:22:36.496380 13751 raft_consensus.cc:2454] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb30c8b777354f6abd0cba489d494d39 in term 0.
I20260621 05:22:36.496380 13886 raft_consensus.cc:2454] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb30c8b777354f6abd0cba489d494d39 in term 0.
I20260621 05:22:36.496675 13548 leader_election.cc:302] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21ce7b6f45294fab8462dbf5bca42d94, bb30c8b777354f6abd0cba489d494d39; no voters:
I20260621 05:22:36.496907 13958 raft_consensus.cc:2781] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:22:36.496977 13958 raft_consensus.cc:482] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:22:36.497014 13958 raft_consensus.cc:3037] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:22:36.498126 13958 raft_consensus.cc:504] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6a0b7372f349fd908ccb851cb72b61" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37205 } } peers { permanent_uuid: "21ce7b6f45294fab8462dbf5bca42d94" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45719 } } peers { permanent_uuid: "bb30c8b777354f6abd0cba489d494d39" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 38077 } }
I20260621 05:22:36.498273 13958 leader_election.cc:288] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39 [CANDIDATE]: Term 1 election: Requested vote from peers 7a6a0b7372f349fd908ccb851cb72b61 (127.2.32.3:37205), 21ce7b6f45294fab8462dbf5bca42d94 (127.2.32.2:45719)
I20260621 05:22:36.498533 13886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8889beb66f374264b1453fe0ebd43066" candidate_uuid: "bb30c8b777354f6abd0cba489d494d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6a0b7372f349fd908ccb851cb72b61"
I20260621 05:22:36.498548 13751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8889beb66f374264b1453fe0ebd43066" candidate_uuid: "bb30c8b777354f6abd0cba489d494d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ce7b6f45294fab8462dbf5bca42d94"
I20260621 05:22:36.498623 13886 raft_consensus.cc:3037] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:22:36.498628 13751 raft_consensus.cc:3037] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:22:36.499461 13886 raft_consensus.cc:2454] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb30c8b777354f6abd0cba489d494d39 in term 1.
I20260621 05:22:36.499581 13751 raft_consensus.cc:2454] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb30c8b777354f6abd0cba489d494d39 in term 1.
I20260621 05:22:36.499656 13548 leader_election.cc:302] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a6a0b7372f349fd908ccb851cb72b61, bb30c8b777354f6abd0cba489d494d39; no voters:
I20260621 05:22:36.499768 13958 raft_consensus.cc:2781] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:22:36.499913 13958 raft_consensus.cc:686] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39 [term 1 LEADER]: Becoming Leader. State: Replica: bb30c8b777354f6abd0cba489d494d39, State: Running, Role: LEADER
I20260621 05:22:36.500007 13958 consensus_queue.cc:237] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6a0b7372f349fd908ccb851cb72b61" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37205 } } peers { permanent_uuid: "21ce7b6f45294fab8462dbf5bca42d94" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45719 } } peers { permanent_uuid: "bb30c8b777354f6abd0cba489d494d39" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 38077 } }
I20260621 05:22:36.500969 13448 catalog_manager.cc:5495] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb30c8b777354f6abd0cba489d494d39 (127.2.32.1). New cstate: current_term: 1 leader_uuid: "bb30c8b777354f6abd0cba489d494d39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6a0b7372f349fd908ccb851cb72b61" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21ce7b6f45294fab8462dbf5bca42d94" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb30c8b777354f6abd0cba489d494d39" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 38077 } health_report { overall_health: HEALTHY } } }
W20260621 05:22:36.512446 13661 tablet.cc:2369] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:22:36.536245 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:22:36.532388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53648 (local address 127.2.32.1:38077)
0621 05:22:36.532436 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:22:36.532438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:22:36.532448 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:22:36.532501 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:22:36.532502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:22:36.532523 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:22:36.532626 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:22:36.532763 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.533466 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.534498 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:22:36.535048 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:22:36.535264 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:22:36.535516 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:22:36.535693 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:22:36.535778 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 05:22:36.536081 (+ 303us) server_negotiation.cc:299] Negotiation successful
0621 05:22:36.536115 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:22:36.538892 13751 raft_consensus.cc:1261] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94 [term 1 FOLLOWER]: Refusing update from remote peer bb30c8b777354f6abd0cba489d494d39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:22:36.538906 13886 raft_consensus.cc:1261] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61 [term 1 FOLLOWER]: Refusing update from remote peer bb30c8b777354f6abd0cba489d494d39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:22:36.539404 13958 consensus_queue.cc:1035] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21ce7b6f45294fab8462dbf5bca42d94" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:22:36.539621 13958 consensus_queue.cc:1035] T 8889beb66f374264b1453fe0ebd43066 P bb30c8b777354f6abd0cba489d494d39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6a0b7372f349fd908ccb851cb72b61" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:22:36.545967 13964 mvcc.cc:205] Tried to move back new op lower bound from 7299151284379144192 to 7299151284224684032. Current Snapshot: MvccSnapshot[applied={T|T < 7299151284379144192}]
W20260621 05:22:36.547500 13799 tablet.cc:2369] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:22:36.548112 13967 mvcc.cc:205] Tried to move back new op lower bound from 7299151284379144192 to 7299151284224684032. Current Snapshot: MvccSnapshot[applied={T|T < 7299151284379144192}]
I20260621 05:22:36.550714 13965 mvcc.cc:205] Tried to move back new op lower bound from 7299151284379144192 to 7299151284224684032. Current Snapshot: MvccSnapshot[applied={T|T < 7299151284379144192}]
I20260621 05:22:36.553112 13448 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:22:36.553220 13448 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:22:36.553956 13448 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a3b8d90ddb54feb87b1bc609de09f0d: Sending DeleteTablet for 3 replicas of tablet 8889beb66f374264b1453fe0ebd43066
I20260621 05:22:36.554272 13594 tablet_service.cc:1514] Processing DeleteTablet for tablet 8889beb66f374264b1453fe0ebd43066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:22:36 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:53634
I20260621 05:22:36.554345 13866 tablet_service.cc:1514] Processing DeleteTablet for tablet 8889beb66f374264b1453fe0ebd43066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:22:36 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:53110
I20260621 05:22:36.554533 13730 tablet_service.cc:1514] Processing DeleteTablet for tablet 8889beb66f374264b1453fe0ebd43066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:22:36 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:51988
I20260621 05:22:36.554819 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 13529
I20260621 05:22:36.559264 13974 tablet_replica.cc:330] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61: stopping tablet replica
I20260621 05:22:36.559412 13975 tablet_replica.cc:330] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94: stopping tablet replica
I20260621 05:22:36.559569 13975 raft_consensus.cc:2229] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:22:36.559685 13975 raft_consensus.cc:2258] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:22:36.559851 13974 raft_consensus.cc:2229] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:22:36.559940 13974 raft_consensus.cc:2258] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:22:36.566011 13975 ts_tablet_manager.cc:1905] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:22:36.566205 13974 ts_tablet_manager.cc:1905] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:22:36.567886 13974 ts_tablet_manager.cc:1918] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:22:36.567970 13974 log.cc:1196] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/8889beb66f374264b1453fe0ebd43066
I20260621 05:22:36.567906 13975 ts_tablet_manager.cc:1918] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:22:36.568535 13974 ts_tablet_manager.cc:1939] T 8889beb66f374264b1453fe0ebd43066 P 7a6a0b7372f349fd908ccb851cb72b61: Deleting consensus metadata
I20260621 05:22:36.568905 13975 log.cc:1196] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/8889beb66f374264b1453fe0ebd43066
I20260621 05:22:36.569038 13435 catalog_manager.cc:4841] TS 7a6a0b7372f349fd908ccb851cb72b61 (127.2.32.3:37205): tablet 8889beb66f374264b1453fe0ebd43066 (table test-table [id=81c300af17e3477388016cff1afa12b9]) successfully deleted
I20260621 05:22:36.571425 13975 ts_tablet_manager.cc:1939] T 8889beb66f374264b1453fe0ebd43066 P 21ce7b6f45294fab8462dbf5bca42d94: Deleting consensus metadata
I20260621 05:22:36.571962 13435 catalog_manager.cc:4841] TS 21ce7b6f45294fab8462dbf5bca42d94 (127.2.32.2:45719): tablet 8889beb66f374264b1453fe0ebd43066 (table test-table [id=81c300af17e3477388016cff1afa12b9]) successfully deleted
W20260621 05:22:36.573071 13433 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.32.1:38077 (error 108)
I20260621 05:22:36.573701 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 13666
I20260621 05:22:36.574311 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:22:36.573785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:38077 (local address 127.0.0.1:53650)
0621 05:22:36.574232 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0621 05:22:36.574271 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.1:38077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425}
W20260621 05:22:36.574481 13433 catalog_manager.cc:4584] TS bb30c8b777354f6abd0cba489d494d39 (127.2.32.1:38077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8889beb66f374264b1453fe0ebd43066: Network error: Client connection negotiation failed: client connection to 127.2.32.1:38077: connect: Connection refused (error 111)
I20260621 05:22:36.580855 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 13803
I20260621 05:22:36.586184 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 13416
I20260621 05:22:36.593178 2176 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260621 05:22:36.938549 2176 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260621 05:22:36.938647 2176 mini_ranger.cc:66] Stopping Ranger...
I20260621 05:22:37.283735 2176 mini_ranger.cc:68] Stopped Ranger
2026-06-21 05:22:37.283 UTC [13002] LOG: received smart shutdown request
2026-06-21 05:22:37.287 UTC [13002] LOG: background worker "logical replication launcher" (PID 13011) exited with exit code 1
2026-06-21 05:22:37.287 UTC [13006] LOG: shutting down
2026-06-21 05:22:37.477 UTC [13002] LOG: database system is shut down
2026-06-21T05:22:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48901 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:22:37 dist-test-slave-b357 krb5kdc[13985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:22:37 dist-test-slave-b357 krb5kdc[13985](info): set up 2 sockets
Jun 21 05:22:37 dist-test-slave-b357 krb5kdc[13985](info): commencing operation
krb5kdc: starting...
W20260621 05:22:38.667519 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:22:38 dist-test-slave-b357 krb5kdc[13985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:22:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:22:38Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.32.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.32.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.32.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/rangeradmin_127.2.32.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.32.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/rangeradmin_127.2.32.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.32.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.32.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.32.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/rangerlookup_127.2.32.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.32.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/rangerlookup_127.2.32.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.32.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.32.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.32.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/HTTP_127.2.32.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.32.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/HTTP_127.2.32.20@KRBTEST.COM.keytab.
I20260621 05:22:38.844069 2176 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskJjmGzv/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/postgres -l logfile start
2026-06-21 05:22:41.593 UTC [14015] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-21 05:22:41.594 UTC [14015] LOG: listening on IPv4 address "127.2.32.20", port 33713
2026-06-21 05:22:41.597 UTC [14015] LOG: listening on Unix socket "/tmp/.s.PGSQL.33713"
2026-06-21 05:22:41.633 UTC [14018] LOG: database system was shut down at 2026-06-21 05:22:39 UTC
2026-06-21 05:22:41.637 UTC [14015] LOG: database system is ready to accept connections
I20260621 05:22:42.595052 2176 mini_postgres.cc:96] Postgres bound to 33713
2026-06-21 05:22:42.602 UTC [14026] FATAL: database "slave" does not exist
127.2.32.20:33713 - accepting connections
I20260621 05:22:42.603673 2176 mini_ranger.cc:161] Starting Ranger...
I20260621 05:22:42.620505 2176 mini_ranger.cc:84] Created miniranger Postgres user
I20260621 05:22:43.168286 2176 mini_ranger.cc:87] Created ranger Postgres database
I20260621 05:22:43.168382 2176 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ranger-admin
2026-06-21 05:22:43,452 [I] DB FLAVOR :POSTGRES
2026-06-21 05:22:43,453 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 05:22:43,453 [I] Checking connection..
2026-06-21 05:22:43,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-21 05:22:43,709 [I] Checking connection passed.
2026-06-21 05:22:43,709 [I] --------- Verifying version history table ---------
2026-06-21 05:22:43,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 05:22:43,988 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 05:22:43,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 05:22:44,250 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 05:22:44,250 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-21 05:22:44,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-21 05:22:44.525 UTC [14090] WARNING: there is no transaction in progress
2026-06-21 05:22:44,532 [I] create_dbversion_catalog.sql file imported successfully
2026-06-21 05:22:44,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 05:22:44,812 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-21 05:22:44,812 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 05:22:44,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-21 05:22:45,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-21 05:22:45,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-b357.c.gcp-upstream.internal','N') ;"
2026-06-21 05:22:45,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 05:22:45,861 [I] Table x_portal_user does not exist in database ranger
2026-06-21 05:22:45,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 05:22:46,115 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-21 05:22:46,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 05:22:46,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 05:22:46,629 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-21 05:22:46,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-21 05:22:47.701 UTC [14198] WARNING: there is no transaction in progress
2026-06-21 05:22:47.719 UTC [14198] WARNING: there is no transaction in progress
2026-06-21 05:22:47.737 UTC [14198] WARNING: there is no transaction in progress
2026-06-21 05:22:47.755 UTC [14198] WARNING: there is no transaction in progress
2026-06-21 05:22:47.777 UTC [14198] WARNING: there is no transaction in progress
2026-06-21 05:22:47.797 UTC [14198] WARNING: there is no transaction in progress
2026-06-21 05:22:47.859 UTC [14198] WARNING: there is no transaction in progress
2026-06-21 05:22:47.867 UTC [14198] WARNING: there is no transaction in progress
2026-06-21 05:22:47.884 UTC [14198] WARNING: there is no transaction in progress
2026-06-21 05:22:47.895 UTC [14198] WARNING: there is no transaction in progress
2026-06-21 05:22:47.908 UTC [14198] WARNING: there is no transaction in progress
2026-06-21 05:22:47.923 UTC [14198] WARNING: there is no transaction in progress
2026-06-21 05:22:47.935 UTC [14198] WARNING: there is no transaction in progress
2026-06-21 05:22:47.943 UTC [14198] WARNING: there is no transaction in progress
2026-06-21 05:22:47.950 UTC [14198] WARNING: there is no transaction in progress
2026-06-21 05:22:47.957 UTC [14198] WARNING: there is no transaction in progress
2026-06-21 05:22:48,712 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-21 05:22:48,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-21 05:22:48,947 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-21 05:22:48,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 05:22:49,229 [I] Table x_portal_user already exists in database 'ranger'
2026-06-21 05:22:49,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 05:22:49,473 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-21 05:22:49,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 05:22:49,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 05:22:50,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-b357.c.gcp-upstream.internal';"
2026-06-21 05:22:50,274 [I] CORE_DB_SCHEMA import status has been updated
2026-06-21 05:22:50,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-21 05:22:50,535 [I] DB_PATCHES have already been applied
I20260621 05:22:50.540763 2176 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ranger-admin:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskJjmGzv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260621 05:22:50.540870 2176 mini_ranger.cc:193] Using host: 127.2.32.20
I20260621 05:22:50.544204 2176 mini_ranger.cc:227] Ranger admin URL: http://127.2.32.20:56909
Jun 21, 2026 5:22:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 5:22:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 5:22:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ranger-admin/ews/webapp
Jun 21, 2026 5:22:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 21, 2026 5:22:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 21, 2026 5:22:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ranger-admin/ews/webapp].
Jun 21, 2026 5:22:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260621 05:22:51.654004 2176 mini_ranger.cc:160] Time spent starting Ranger: real 9.050s user 0.003s sys 0.007s
Jun 21, 2026 5:23:12 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 21, 2026 5:23:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 21, 2026 5:23:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 21, 2026 5:23:12 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 21, 2026 5:23:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 21, 2026 5:23:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 21, 2026 5:23:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 21, 2026 5:23:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 21, 2026 5:23:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 21, 2026 5:23:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 21, 2026 5:23:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 21, 2026 5:23:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 21, 2026 5:23:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 21, 2026 5:23:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 21, 2026 5:23:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 21, 2026 5:23:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 21, 2026 5:23:12 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 21, 2026 5:23:13 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260621 05:23:14.205678 2176 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.32.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.32.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.32.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/rangerkms_127.2.32.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.32.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/rangerkms_127.2.32.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.32.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.32.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.32.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/HTTP_127.2.32.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.32.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/HTTP_127.2.32.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 21 05:23:14 dist-test-slave-b357 krb5kdc[13985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019394, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260621 05:23:14.288120 2176 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260621 05:23:14.298434 2176 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260621 05:23:15.255625 2176 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260621 05:23:15.255718 2176 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ranger-kms
I20260621 05:23:15.255735 2176 mini_ranger_kms.cc:227] Using postgres at 127.2.32.20:33713
2026-06-21 05:23:15,464 [I] DB FLAVOR :POSTGRES
2026-06-21 05:23:15,465 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 05:23:15,465 [I] Checking connection
2026-06-21 05:23:15,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-21 05:23:15,717 [I] connection success
2026-06-21 05:23:15,718 [I] --------- Verifying Ranger DB tables ---------
2026-06-21 05:23:15,718 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-21 05:23:15,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-21 05:23:15,983 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-21 05:23:15,984 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 05:23:15,984 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-21 05:23:15,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.32.20:33713/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-21 05:23:16,269 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260621 05:23:16.826417 2176 mini_ranger_kms.cc:310] Created kms service
I20260621 05:23:17.059772 2176 mini_ranger_kms.cc:326] Created kudu user
I20260621 05:23:17.106520 2176 mini_ranger_kms.cc:343] Created rangerkms user
Jun 21 05:23:17 dist-test-slave-b357 krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019394, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
I20260621 05:23:17.554580 2176 mini_ranger_kms.cc:384] Added ranger policy
I20260621 05:23:17.554821 2176 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ranger-kms:/tmp/dist-test-taskJjmGzv/build/release/bin/postgresql.jar:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskJjmGzv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260621 05:23:17.554843 2176 mini_ranger_kms.cc:240] Using host: 127.2.32.20
I20260621 05:23:17.558393 2176 mini_ranger_kms.cc:276] Ranger KMS PID: 14379
I20260621 05:23:17.558470 2176 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.32.20:35677
Jun 21, 2026 5:23:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 5:23:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 5:23:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 21, 2026 5:23:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 21, 2026 5:23:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ranger-kms/ews/webapp].
Jun 21, 2026 5:23:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260621 05:23:19.895336 2176 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.607s user 0.006s sys 0.186s
I20260621 05:23:19.895458 2176 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260621 05:23:19.895512 2176 mini_ranger_kms.cc:401] 127.2.32.20:35677/kms/v1/keys
Jun 21 05:23:20 dist-test-slave-b357 krb5kdc[13985](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782019400, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.32.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 21 05:23:20 dist-test-slave-b357 krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019400, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.32.20@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
Jun 21 05:23:21 dist-test-slave-b357 krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019400, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.32.20@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
Jun 21 05:23:23 dist-test-slave-b357 krb5kdc[13985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:23:23.432192 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:43547
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:43547
--ranger_config_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.32.20:35677/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:23:23.514526 14429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:23.514734 14429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:23.515017 14429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:23.517102 14429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:23:23.517163 14429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:23.517180 14429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:23.517194 14429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:23:23.517210 14429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:23:23.519425 14429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.32.20:35677/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:43547
--ranger_config_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:43547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:23.519805 14429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:23.520107 14429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:23.526207 14435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:23.526221 14437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:23.526332 14429 server_base.cc:1034] running on GCE node
W20260621 05:23:23.526418 14434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:23.526993 14429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:23.527367 14429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:23.528546 14429 hybrid_clock.cc:648] HybridClock initialized: now 1782019403528538 us; error 38 us; skew 500 ppm
Jun 21 05:23:23 dist-test-slave-b357 krb5kdc[13985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019403, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:23.531885 14429 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:23:23.532605 14429 webserver.cc:460] Webserver started at http://127.2.32.62:39887/ using document root <none> and password file <none>
I20260621 05:23:23.532876 14429 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:23.532932 14429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:23.533095 14429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:23:23 dist-test-slave-b357 krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019403, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
I20260621 05:23:23.585106 14429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "1f10936fcc2049c1aae31332d92af20b"
format_stamp: "Formatted at 2026-06-21 05:23:23 on dist-test-slave-b357"
server_key: "8170e7685a462fd72fe3acf07734fa8f"
server_key_iv: "3d6edd6e565400640dbac3dfd571d58c"
server_key_version: "kuduclusterkey@0"
I20260621 05:23:23.585592 14429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "1f10936fcc2049c1aae31332d92af20b"
format_stamp: "Formatted at 2026-06-21 05:23:23 on dist-test-slave-b357"
server_key: "8170e7685a462fd72fe3acf07734fa8f"
server_key_iv: "3d6edd6e565400640dbac3dfd571d58c"
server_key_version: "kuduclusterkey@0"
I20260621 05:23:23.587080 14429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260621 05:23:23.599852 14444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:23.600119 14429 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:23:23.600234 14429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "1f10936fcc2049c1aae31332d92af20b"
format_stamp: "Formatted at 2026-06-21 05:23:23 on dist-test-slave-b357"
server_key: "8170e7685a462fd72fe3acf07734fa8f"
server_key_iv: "3d6edd6e565400640dbac3dfd571d58c"
server_key_version: "kuduclusterkey@0"
I20260621 05:23:23.600369 14429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:23.614532 14429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:23.616796 14429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:23.616951 14429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:23.621912 14429 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:43547
I20260621 05:23:23.621976 14496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:43547 every 8 connection(s)
I20260621 05:23:23.622341 14429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:23:23.623026 14497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:23:23.625298 14497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b: Bootstrap starting.
I20260621 05:23:23.625883 14497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b: Neither blocks nor log segments found. Creating new log.
I20260621 05:23:23.626158 14497 log.cc:824] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:23.626870 14497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b: No bootstrap required, opened a new log
I20260621 05:23:23.628306 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 14429
I20260621 05:23:23.628407 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:23:23.628612 2176 external_mini_cluster.cc:1428] Setting key ab5acd42706c05fd05c986da5d1ed0a5
I20260621 05:23:23.629348 14497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f10936fcc2049c1aae31332d92af20b" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43547 } }
I20260621 05:23:23.629530 14497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:23:23.629565 14497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10936fcc2049c1aae31332d92af20b, State: Initialized, Role: FOLLOWER
I20260621 05:23:23.629735 14497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f10936fcc2049c1aae31332d92af20b" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43547 } }
I20260621 05:23:23.629860 14497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:23:23.629917 14497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:23:23.629971 14497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:23:23.630509 14497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f10936fcc2049c1aae31332d92af20b" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43547 } }
I20260621 05:23:23.630620 14497 leader_election.cc:302] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f10936fcc2049c1aae31332d92af20b; no voters:
I20260621 05:23:23.630774 14497 leader_election.cc:288] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:23:23.630836 14502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:23:23.631043 14502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [term 1 LEADER]: Becoming Leader. State: Replica: 1f10936fcc2049c1aae31332d92af20b, State: Running, Role: LEADER
I20260621 05:23:23.631071 14497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:23:23.631206 14502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f10936fcc2049c1aae31332d92af20b" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43547 } }
I20260621 05:23:23.631574 14502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f10936fcc2049c1aae31332d92af20b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f10936fcc2049c1aae31332d92af20b" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43547 } } }
I20260621 05:23:23.631714 14502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [sys.catalog]: This master's current role is: LEADER
I20260621 05:23:23.631887 14502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f10936fcc2049c1aae31332d92af20b. Latest consensus state: current_term: 1 leader_uuid: "1f10936fcc2049c1aae31332d92af20b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f10936fcc2049c1aae31332d92af20b" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43547 } } }
I20260621 05:23:23.631987 14502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [sys.catalog]: This master's current role is: LEADER
I20260621 05:23:23.632146 14511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:23:23.633270 14511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:23:23.633546 14497 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 05:23:23.635198 14511 catalog_manager.cc:1347] Generated new cluster ID: 66db586d0bf1409e8cd5173da91d23f1
I20260621 05:23:23.635299 14511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:23:23 dist-test-slave-b357 krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:23.641096 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:23.629176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56812 (local address 127.2.32.62:43547)
0621 05:23:23.629369 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:23.629379 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:23.629410 (+ 31us) server_negotiation.cc:407] Connection header received
0621 05:23:23.629521 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:23.629528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:23.629591 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:23.629727 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:23.630780 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:23.632383 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:23.633538 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:23.634173 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:23.634281 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:23.637404 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:23.637422 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:23.637424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:23.637448 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:23.639440 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:23.639853 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:23.639856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:23.639857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:23.639981 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:23.640194 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:23.640194 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:23.640195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:23.640491 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:23.640649 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:23.640771 (+ 122us) server_negotiation.cc:299] Negotiation successful
0621 05:23:23.640894 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":82,"threads_started":1}
I20260621 05:23:23.657100 14511 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:23:23.657953 14511 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:23:23.664687 14511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b: Generated new TSK 0
I20260621 05:23:23.664938 14511 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
I20260621 05:23:24.374482 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.32.20:35677/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:43547
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:23:24.483624 14542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:24.483973 14542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:24.484028 14542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:24.485880 14542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:24.485980 14542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:24.486035 14542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:23:24.488832 14542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.32.20:35677/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:43547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:24.489151 14542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:24.489389 14542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:24.494434 14550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:24.494589 14547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:24.494451 14548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:24.494815 14542 server_base.cc:1034] running on GCE node
I20260621 05:23:24.495054 14542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:24.495362 14542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:24.496548 14542 hybrid_clock.cc:648] HybridClock initialized: now 1782019404496534 us; error 65 us; skew 500 ppm
Jun 21 05:23:24 dist-test-slave-b357 krb5kdc[13985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019404, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:24.499850 14542 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:23:24.500607 14542 webserver.cc:460] Webserver started at http://127.2.32.1:41015/ using document root <none> and password file <none>
I20260621 05:23:24.500917 14542 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:24.500988 14542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:24.501129 14542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:23:24 dist-test-slave-b357 krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019404, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
I20260621 05:23:24.518236 14542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "0382548e9bed473488d574fa766e9bf2"
format_stamp: "Formatted at 2026-06-21 05:23:24 on dist-test-slave-b357"
server_key: "02ddc9258b8b6966e97c862049367dfc"
server_key_iv: "84d5d3961c8fac7add88e80d6e6d4cc3"
server_key_version: "kuduclusterkey@0"
I20260621 05:23:24.518718 14542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "0382548e9bed473488d574fa766e9bf2"
format_stamp: "Formatted at 2026-06-21 05:23:24 on dist-test-slave-b357"
server_key: "02ddc9258b8b6966e97c862049367dfc"
server_key_iv: "84d5d3961c8fac7add88e80d6e6d4cc3"
server_key_version: "kuduclusterkey@0"
I20260621 05:23:24.520695 14542 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260621 05:23:24.535264 14559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:24.535445 14542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 05:23:24.535545 14542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "0382548e9bed473488d574fa766e9bf2"
format_stamp: "Formatted at 2026-06-21 05:23:24 on dist-test-slave-b357"
server_key: "02ddc9258b8b6966e97c862049367dfc"
server_key_iv: "84d5d3961c8fac7add88e80d6e6d4cc3"
server_key_version: "kuduclusterkey@0"
I20260621 05:23:24.535648 14542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:24.554992 14542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:24.557011 14542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:24.557168 14542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:24.557390 14542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:23:24.557776 14542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:23:24.557814 14542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:24.557852 14542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:23:24.557878 14542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:24.567500 14542 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:40505
I20260621 05:23:24.568001 14542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:23:24.568449 14672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:40505 every 8 connection(s)
I20260621 05:23:24.575564 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 14542
I20260621 05:23:24.575662 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:23:24.575858 2176 external_mini_cluster.cc:1428] Setting key 28f7e30fa1a1434cc356ac0a631c57d6
Jun 21 05:23:24 dist-test-slave-b357 krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019404, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:24.581040 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:24.569247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:42635 (local address 127.2.32.62:43547)
0621 05:23:24.571362 (+ 2115us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:24.571364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:24.571618 (+ 254us) server_negotiation.cc:407] Connection header received
0621 05:23:24.572175 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:24.572176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:24.572191 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:24.572266 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:24.572841 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:24.573365 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:24.574340 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:24.574549 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:24.574623 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:24.577149 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:24.577150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:24.577151 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:24.577158 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:24.579054 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:24.579486 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:24.579488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:24.579488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:24.579549 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:24.580034 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:24.580037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:24.580037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:24.580247 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:24.580340 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:24.580778 (+ 438us) server_negotiation.cc:299] Negotiation successful
0621 05:23:24.580915 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2065,"thread_start_us":120,"threads_started":1}
I20260621 05:23:24.581215 14675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:24.569224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:43547 (local address 127.2.32.1:42635)
0621 05:23:24.571447 (+ 2223us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:24.571495 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:24.572065 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:24.572338 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:24.572343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:24.572543 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:24.572747 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:24.572751 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:24.573456 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:24.573457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:24.574244 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:24.574246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:24.574661 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:24.574662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:24.574779 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:24.575345 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:24.575367 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:24.577052 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:24.579162 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:24.579169 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:24.579171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:24.579398 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:24.579617 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:24.579617 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:24.579618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:24.579729 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:24.580330 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:24.580345 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:24.580603 (+ 258us) client_negotiation.cc:769] Sending connection context
0621 05:23:24.580730 (+ 127us) client_negotiation.cc:240] Negotiation successful
0621 05:23:24.580929 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2173,"thread_start_us":131,"threads_started":1}
I20260621 05:23:24.582192 14673 heartbeater.cc:347] Connected to a master server at 127.2.32.62:43547
I20260621 05:23:24.582381 14673 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:24.582887 14673 heartbeater.cc:511] Master 127.2.32.62:43547 requested a full tablet report, sending...
I20260621 05:23:24.583803 14461 ts_manager.cc:194] Registered new tserver with Master: 0382548e9bed473488d574fa766e9bf2 (127.2.32.1:40505)
I20260621 05:23:24.584746 14461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:42635
WARNING: no policy specified for kudu/127.2.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
I20260621 05:23:24.607872 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.32.20:35677/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:43547
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 05:23:24 dist-test-slave-b357 krb5kdc[13985](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782019404, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260621 05:23:24.729220 14679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:24.729425 14679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:24.729472 14679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:24.730602 14679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:24.730661 14679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:24.730715 14679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:23:24.732812 14679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.32.20:35677/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:43547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:24.733124 14679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:24.733374 14679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:24.738883 14689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:24.739804 14687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:24.740052 14686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:24.740752 14679 server_base.cc:1034] running on GCE node
I20260621 05:23:24.740962 14679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:24.741218 14679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:24.744561 14679 hybrid_clock.cc:648] HybridClock initialized: now 1782019404744525 us; error 48 us; skew 500 ppm
Jun 21 05:23:24 dist-test-slave-b357 krb5kdc[13985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019404, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:24.747856 14679 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:23:24.748445 14679 webserver.cc:460] Webserver started at http://127.2.32.2:42985/ using document root <none> and password file <none>
I20260621 05:23:24.748718 14679 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:24.748790 14679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:24.748953 14679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:23:24 dist-test-slave-b357 krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019404, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
I20260621 05:23:24.768571 14679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "ee8536f60ee24a8f8495b917773101f3"
format_stamp: "Formatted at 2026-06-21 05:23:24 on dist-test-slave-b357"
server_key: "9f3481f85e97163908ef19f962e685c6"
server_key_iv: "f5f6c2c35683e4a34772b25b217c9625"
server_key_version: "kuduclusterkey@0"
I20260621 05:23:24.769079 14679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "ee8536f60ee24a8f8495b917773101f3"
format_stamp: "Formatted at 2026-06-21 05:23:24 on dist-test-slave-b357"
server_key: "9f3481f85e97163908ef19f962e685c6"
server_key_iv: "f5f6c2c35683e4a34772b25b217c9625"
server_key_version: "kuduclusterkey@0"
I20260621 05:23:24.772640 14679 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260621 05:23:24.784126 14696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:24.784294 14679 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:23:24.784371 14679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "ee8536f60ee24a8f8495b917773101f3"
format_stamp: "Formatted at 2026-06-21 05:23:24 on dist-test-slave-b357"
server_key: "9f3481f85e97163908ef19f962e685c6"
server_key_iv: "f5f6c2c35683e4a34772b25b217c9625"
server_key_version: "kuduclusterkey@0"
I20260621 05:23:24.784454 14679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:24.808591 14679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:24.811050 14679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:24.811234 14679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:24.811478 14679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:23:24.811836 14679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:23:24.811882 14679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:24.811915 14679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:23:24.811935 14679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:24.818935 14679 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:38179
I20260621 05:23:24.820251 14679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:23:24.821439 14810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:38179 every 8 connection(s)
I20260621 05:23:24.826304 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 14679
I20260621 05:23:24.826396 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:23:24.826617 2176 external_mini_cluster.cc:1428] Setting key b51eabd274bd3c1322c533d348ccafec
Jun 21 05:23:24 dist-test-slave-b357 krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019404, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:24.833014 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:24.822381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:53865 (local address 127.2.32.62:43547)
0621 05:23:24.822438 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:24.822441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:24.822847 (+ 406us) server_negotiation.cc:407] Connection header received
0621 05:23:24.823557 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:24.823559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:24.823576 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:24.823667 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:24.824409 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:24.825292 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:24.826784 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:24.827115 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:24.827188 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:24.829757 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:24.829758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:24.829759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:24.829766 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:24.831481 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:24.831942 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:24.831944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:24.831944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:24.832001 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:24.832264 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:24.832265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:24.832266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:24.832410 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:24.832455 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:24.832825 (+ 370us) server_negotiation.cc:299] Negotiation successful
0621 05:23:24.832914 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:23:24.833529 14813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:24.822389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:43547 (local address 127.2.32.2:53865)
0621 05:23:24.822650 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:24.822701 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:24.823409 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:24.823759 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:24.823764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:24.824015 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:24.824289 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:24.824295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:24.825415 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:24.825416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:24.826675 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:24.826678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:24.827293 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:24.827294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:24.827454 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:24.828068 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:24.828078 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:24.829649 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:24.831570 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:24.831576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:24.831579 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:24.831852 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:24.832070 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:24.832070 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:24.832071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:24.832189 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:24.832503 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:24.832509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:24.832678 (+ 169us) client_negotiation.cc:769] Sending connection context
0621 05:23:24.833139 (+ 461us) client_negotiation.cc:240] Negotiation successful
0621 05:23:24.833293 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":145,"threads_started":1}
I20260621 05:23:24.834147 14811 heartbeater.cc:347] Connected to a master server at 127.2.32.62:43547
I20260621 05:23:24.834244 14811 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:24.834436 14811 heartbeater.cc:511] Master 127.2.32.62:43547 requested a full tablet report, sending...
I20260621 05:23:24.834795 14461 ts_manager.cc:194] Registered new tserver with Master: ee8536f60ee24a8f8495b917773101f3 (127.2.32.2:38179)
I20260621 05:23:24.835306 14461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:53865
WARNING: no policy specified for kudu/127.2.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
I20260621 05:23:24.859591 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.32.20:35677/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:43547
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:23:24.985915 14816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:24.986198 14816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:24.986238 14816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:24.988373 14816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:24.988473 14816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:24.988546 14816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:23:24.990752 14816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.32.20:35677/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:43547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:24.991014 14816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:24.991308 14816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:24.997355 14822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:24.997529 14824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:24.997922 14821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:24.998601 14816 server_base.cc:1034] running on GCE node
I20260621 05:23:24.998971 14816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:24.999349 14816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:25.000627 14816 hybrid_clock.cc:648] HybridClock initialized: now 1782019405000599 us; error 41 us; skew 500 ppm
Jun 21 05:23:25 dist-test-slave-b357 krb5kdc[13985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019405, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:25.003973 14816 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:23:25.004437 14816 webserver.cc:460] Webserver started at http://127.2.32.3:33575/ using document root <none> and password file <none>
I20260621 05:23:25.004760 14816 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:25.004827 14816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:25.004966 14816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 05:23:25 dist-test-slave-b357 krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019405, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
I20260621 05:23:25.018064 14816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "1c665e912d1040c2b8ad28e9200f8dbc"
format_stamp: "Formatted at 2026-06-21 05:23:25 on dist-test-slave-b357"
server_key: "528c5a10ea6bcfc54e0912a49aa38369"
server_key_iv: "6c184598c0f53809a34c09b85cc3f899"
server_key_version: "kuduclusterkey@0"
I20260621 05:23:25.018512 14816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "1c665e912d1040c2b8ad28e9200f8dbc"
format_stamp: "Formatted at 2026-06-21 05:23:25 on dist-test-slave-b357"
server_key: "528c5a10ea6bcfc54e0912a49aa38369"
server_key_iv: "6c184598c0f53809a34c09b85cc3f899"
server_key_version: "kuduclusterkey@0"
I20260621 05:23:25.021003 14816 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260621 05:23:25.035267 14831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:25.035509 14816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260621 05:23:25.035593 14816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "1c665e912d1040c2b8ad28e9200f8dbc"
format_stamp: "Formatted at 2026-06-21 05:23:25 on dist-test-slave-b357"
server_key: "528c5a10ea6bcfc54e0912a49aa38369"
server_key_iv: "6c184598c0f53809a34c09b85cc3f899"
server_key_version: "kuduclusterkey@0"
I20260621 05:23:25.035835 14816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:25.070802 14816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:25.073137 14816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:25.073287 14816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:25.073544 14816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:23:25.073899 14816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:23:25.073949 14816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:25.073983 14816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:23:25.074007 14816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:25.081831 14816 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:35453
I20260621 05:23:25.082266 14816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:23:25.082863 14944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:35453 every 8 connection(s)
Jun 21 05:23:25 dist-test-slave-b357 krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019405, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:25.090142 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 14816
I20260621 05:23:25.090252 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:23:25.090449 2176 external_mini_cluster.cc:1428] Setting key 78a6703ac041e5ef6423388eb089a943
I20260621 05:23:25.094461 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:25.083259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:44241 (local address 127.2.32.62:43547)
0621 05:23:25.083343 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:25.083346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:25.083582 (+ 236us) server_negotiation.cc:407] Connection header received
0621 05:23:25.084255 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:25.084256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:25.084268 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:25.084335 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:25.085047 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:25.085806 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:25.087282 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:25.087595 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:25.087641 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:25.090274 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:25.090276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:25.090277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:25.090284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:25.092430 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:25.093328 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:25.093329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:25.093329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:25.093387 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:25.093687 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:25.093688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:25.093688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:25.093819 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:25.093863 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:25.094272 (+ 409us) server_negotiation.cc:299] Negotiation successful
0621 05:23:25.094366 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 05:23:25.094607 14947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:25.083190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:43547 (local address 127.2.32.3:44241)
0621 05:23:25.083449 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:25.083489 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:25.084121 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:25.084407 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:25.084411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:25.084645 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:25.084933 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:25.084939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:25.085935 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:25.085936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:25.087181 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:25.087184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:25.087743 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:25.087744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:25.087894 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:25.088469 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:25.088479 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:25.090124 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:25.092936 (+ 2812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:25.092942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:25.092944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:25.093233 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:25.093461 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:25.093461 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:25.093462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:25.093602 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:25.093928 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:25.093935 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:25.094102 (+ 167us) client_negotiation.cc:769] Sending connection context
0621 05:23:25.094226 (+ 124us) client_negotiation.cc:240] Negotiation successful
0621 05:23:25.094379 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":92,"threads_started":1}
I20260621 05:23:25.095028 14945 heartbeater.cc:347] Connected to a master server at 127.2.32.62:43547
I20260621 05:23:25.095155 14945 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:25.095353 14945 heartbeater.cc:511] Master 127.2.32.62:43547 requested a full tablet report, sending...
I20260621 05:23:25.096071 14461 ts_manager.cc:194] Registered new tserver with Master: 1c665e912d1040c2b8ad28e9200f8dbc (127.2.32.3:35453)
I20260621 05:23:25.096478 14461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:44241
I20260621 05:23:25.103838 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:23:25.104809 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 14542
I20260621 05:23:25.112430 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 14679
I20260621 05:23:25.118149 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 14816
I20260621 05:23:25.124465 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 14429
I20260621 05:23:25.131476 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:43547
--webserver_interface=127.2.32.62
--webserver_port=39887
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:43547
--ranger_config_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.32.20:35677/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.32.20:35677/kms with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:23:25.214941 14948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:25.215174 14948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:25.215212 14948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:25.216353 14948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:23:25.216403 14948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:25.216418 14948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:25.216430 14948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:23:25.216444 14948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:23:25.217955 14948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.32.20:35677/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:43547
--ranger_config_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:43547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=39887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:25.218428 14948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:25.218827 14948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:25.224037 14953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:25.224104 14954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:25.224108 14956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:25.224365 14948 server_base.cc:1034] running on GCE node
I20260621 05:23:25.224654 14948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:25.224988 14948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:25.226177 14948 hybrid_clock.cc:648] HybridClock initialized: now 1782019405226152 us; error 52 us; skew 500 ppm
Jun 21 05:23:25 dist-test-slave-b357 krb5kdc[13985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019405, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:25.229066 14948 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:23:25.229687 14948 webserver.cc:460] Webserver started at http://127.2.32.62:39887/ using document root <none> and password file <none>
I20260621 05:23:25.229938 14948 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:25.230017 14948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 05:23:25 dist-test-slave-b357 krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019405, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
I20260621 05:23:25.312172 14948 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:25.313122 14964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:25.313335 14948 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:25.313436 14948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "1f10936fcc2049c1aae31332d92af20b"
format_stamp: "Formatted at 2026-06-21 05:23:23 on dist-test-slave-b357"
server_key: "8170e7685a462fd72fe3acf07734fa8f"
server_key_iv: "3d6edd6e565400640dbac3dfd571d58c"
server_key_version: "kuduclusterkey@0"
I20260621 05:23:25.313815 14948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:25.338845 14948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:25.341589 14948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:25.341807 14948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:25.347280 14948 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:43547
I20260621 05:23:25.347325 15016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:43547 every 8 connection(s)
I20260621 05:23:25.347795 14948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:23:25.348230 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 14948
I20260621 05:23:25.348546 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.32.20:35677/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:40505
--local_ip_for_outbound_sockets=127.2.32.1
--tserver_master_addrs=127.2.32.62:43547
--webserver_port=41015
--webserver_interface=127.2.32.1
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.32.20:35677/kms with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:23:25.349427 15017 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:23:25.350402 15017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b: Bootstrap starting.
I20260621 05:23:25.359020 15017 log.cc:824] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:25.361251 15017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:23:25.361637 15017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b: Bootstrap complete.
I20260621 05:23:25.364553 15017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f10936fcc2049c1aae31332d92af20b" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43547 } }
I20260621 05:23:25.364841 15017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10936fcc2049c1aae31332d92af20b, State: Initialized, Role: FOLLOWER
I20260621 05:23:25.365033 15017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f10936fcc2049c1aae31332d92af20b" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43547 } }
I20260621 05:23:25.365113 15017 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:23:25.365167 15017 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:23:25.365217 15017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [term 1 FOLLOWER]: Advancing to term 2
I20260621 05:23:25.366387 15017 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f10936fcc2049c1aae31332d92af20b" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43547 } }
I20260621 05:23:25.366515 15017 leader_election.cc:302] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f10936fcc2049c1aae31332d92af20b; no voters:
I20260621 05:23:25.366760 15017 leader_election.cc:288] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 05:23:25.366807 15021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [term 2 FOLLOWER]: Leader election won for term 2
I20260621 05:23:25.366932 15021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [term 2 LEADER]: Becoming Leader. State: Replica: 1f10936fcc2049c1aae31332d92af20b, State: Running, Role: LEADER
I20260621 05:23:25.367062 15021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f10936fcc2049c1aae31332d92af20b" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43547 } }
I20260621 05:23:25.367152 15017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:23:25.367398 15022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1f10936fcc2049c1aae31332d92af20b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f10936fcc2049c1aae31332d92af20b" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43547 } } }
I20260621 05:23:25.367515 15022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [sys.catalog]: This master's current role is: LEADER
I20260621 05:23:25.367398 15023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f10936fcc2049c1aae31332d92af20b. Latest consensus state: current_term: 2 leader_uuid: "1f10936fcc2049c1aae31332d92af20b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f10936fcc2049c1aae31332d92af20b" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43547 } } }
I20260621 05:23:25.367789 15023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b [sys.catalog]: This master's current role is: LEADER
I20260621 05:23:25.367899 15025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:23:25.368459 15025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:23:25.368615 15025 catalog_manager.cc:1259] Loaded cluster ID: 66db586d0bf1409e8cd5173da91d23f1
I20260621 05:23:25.368665 15025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:23:25.368935 15017 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 05:23:25.372987 15025 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:23:25.373594 15025 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b: Loaded TSK: 0
I20260621 05:23:25.373942 15025 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 05:23:25.434579 15019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:25.434785 15019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:25.434823 15019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:25.435987 15019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:25.436049 15019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:25.436101 15019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:23:25.437528 15019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.32.20:35677/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:40505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=41015
--tserver_master_addrs=127.2.32.62:43547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:25.437768 15019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:25.437968 15019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:25.443114 15052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:25.443163 15049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:25.443114 15050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:25.444248 15019 server_base.cc:1034] running on GCE node
I20260621 05:23:25.444527 15019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:25.444825 15019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:25.448093 15019 hybrid_clock.cc:648] HybridClock initialized: now 1782019405448069 us; error 59 us; skew 500 ppm
Jun 21 05:23:25 dist-test-slave-b357 krb5kdc[13985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019405, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:25.450816 15019 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:23:25.451372 15019 webserver.cc:460] Webserver started at http://127.2.32.1:41015/ using document root <none> and password file <none>
I20260621 05:23:25.451622 15019 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:25.451704 15019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 05:23:25 dist-test-slave-b357 krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019405, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
I20260621 05:23:25.478029 15019 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.000s
I20260621 05:23:25.478866 15065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:25.479027 15019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:25.479171 15019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "0382548e9bed473488d574fa766e9bf2"
format_stamp: "Formatted at 2026-06-21 05:23:24 on dist-test-slave-b357"
server_key: "02ddc9258b8b6966e97c862049367dfc"
server_key_iv: "84d5d3961c8fac7add88e80d6e6d4cc3"
server_key_version: "kuduclusterkey@0"
I20260621 05:23:25.480473 15019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:25.513120 15019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:25.515544 15019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:25.515709 15019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:25.515933 15019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:23:25.516284 15019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:23:25.516330 15019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:25.516369 15019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:23:25.516393 15019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:25.526889 15019 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:40505
I20260621 05:23:25.527381 15019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:23:25.527998 15178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:40505 every 8 connection(s)
I20260621 05:23:25.537541 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 15019
Jun 21 05:23:25 dist-test-slave-b357 krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019405, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:25.537847 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.32.20:35677/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:38179
--local_ip_for_outbound_sockets=127.2.32.2
--tserver_master_addrs=127.2.32.62:43547
--webserver_port=42985
--webserver_interface=127.2.32.2
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.32.20:35677/kms with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:23:25.544373 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:25.528583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:43547 (local address 127.2.32.1:46615)
0621 05:23:25.529886 (+ 1303us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:25.529929 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:25.530663 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:25.531372 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:25.531376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:25.531613 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:25.531875 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:25.531882 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:25.533494 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:25.533495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:25.534778 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:25.534781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:25.535733 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:25.535735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:25.535867 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:25.536469 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:25.536479 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:25.539511 (+ 3032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:25.542139 (+ 2628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:25.542156 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:25.542159 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:25.542647 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:25.543020 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:25.543022 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:25.543022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:25.543196 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:25.543691 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:25.543697 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:25.543858 (+ 161us) client_negotiation.cc:769] Sending connection context
0621 05:23:25.543969 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 05:23:25.544139 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1266,"thread_start_us":84,"threads_started":1}
I20260621 05:23:25.545687 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:25.528683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:46615 (local address 127.2.32.62:43547)
0621 05:23:25.528896 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:25.528905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:25.530845 (+ 1940us) server_negotiation.cc:407] Connection header received
0621 05:23:25.530985 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:25.530991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:25.531042 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:25.531208 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:25.532006 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:25.533382 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:25.534879 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:25.535521 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:25.535625 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:25.539647 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:25.539651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:25.539651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:25.539663 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:25.541971 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:25.542799 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:25.542802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:25.542803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:25.542913 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:25.543278 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:25.543279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:25.543280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:25.543578 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:25.543660 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:25.545104 (+ 1444us) server_negotiation.cc:299] Negotiation successful
0621 05:23:25.545300 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
I20260621 05:23:25.547189 15179 heartbeater.cc:347] Connected to a master server at 127.2.32.62:43547
I20260621 05:23:25.547291 15179 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:25.547511 15179 heartbeater.cc:511] Master 127.2.32.62:43547 requested a full tablet report, sending...
W20260621 05:23:25.548501 15179 heartbeater.cc:644] Failed to heartbeat to 127.2.32.62:43547 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 05:23:25.548568 15179 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:25.548624 15179 heartbeater.cc:511] Master 127.2.32.62:43547 requested a full tablet report, sending...
I20260621 05:23:25.549058 15179 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:25.549129 15179 heartbeater.cc:511] Master 127.2.32.62:43547 requested a full tablet report, sending...
W20260621 05:23:25.549472 15179 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 05:23:25.660473 15183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:25.660751 15183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:25.660790 15183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:25.662513 15183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:25.662570 15183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:25.662611 15183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:23:25.664624 15183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.32.20:35677/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:38179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=42985
--tserver_master_addrs=127.2.32.62:43547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:25.664877 15183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:25.665122 15183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:25.670892 15189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:25.670946 15188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:25.671212 15183 server_base.cc:1034] running on GCE node
W20260621 05:23:25.670975 15191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:25.671547 15183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:25.671793 15183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:25.679212 15183 hybrid_clock.cc:648] HybridClock initialized: now 1782019405679164 us; error 56 us; skew 500 ppm
Jun 21 05:23:25 dist-test-slave-b357 krb5kdc[13985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019405, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:25.682114 15183 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:23:25.682617 15183 webserver.cc:460] Webserver started at http://127.2.32.2:42985/ using document root <none> and password file <none>
I20260621 05:23:25.682827 15183 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:25.682873 15183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 05:23:25 dist-test-slave-b357 krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019405, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
I20260621 05:23:25.702313 15183 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260621 05:23:25.703336 15199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:25.703491 15183 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:25.703554 15183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "ee8536f60ee24a8f8495b917773101f3"
format_stamp: "Formatted at 2026-06-21 05:23:24 on dist-test-slave-b357"
server_key: "9f3481f85e97163908ef19f962e685c6"
server_key_iv: "f5f6c2c35683e4a34772b25b217c9625"
server_key_version: "kuduclusterkey@0"
I20260621 05:23:25.704005 15183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:25.715147 15183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:25.718286 15183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:25.718472 15183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:25.718856 15183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:23:25.719381 15183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:23:25.719442 15183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:25.719478 15183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:23:25.719506 15183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:25.729197 15183 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:38179
I20260621 05:23:25.729267 15312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:38179 every 8 connection(s)
I20260621 05:23:25.729688 15183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:23:25.730000 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 15183
I20260621 05:23:25.730327 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.32.20:35677/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:35453
--local_ip_for_outbound_sockets=127.2.32.3
--tserver_master_addrs=127.2.32.62:43547
--webserver_port=33575
--webserver_interface=127.2.32.3
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.32.20:35677/kms with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 05:23:25 dist-test-slave-b357 krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019405, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:25.743162 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:25.730707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:40189 (local address 127.2.32.62:43547)
0621 05:23:25.730759 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:25.730762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:25.731066 (+ 304us) server_negotiation.cc:407] Connection header received
0621 05:23:25.731922 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:25.731923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:25.731939 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:25.732000 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:25.733759 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:25.734592 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:25.736144 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:25.736500 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:25.736579 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:25.739481 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:25.739483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:25.739483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:25.739491 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:25.741360 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:25.741915 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:25.741916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:25.741917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:25.741979 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:25.742296 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:25.742297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:25.742297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:25.742441 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:25.742498 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:25.742900 (+ 402us) server_negotiation.cc:299] Negotiation successful
0621 05:23:25.742998 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:23:25.743881 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:25.730741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:43547 (local address 127.2.32.2:40189)
0621 05:23:25.730929 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:25.730976 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:25.731782 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:25.732369 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:25.732373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:25.733176 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:25.733655 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:25.733662 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:25.734711 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:25.734713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:25.736044 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:25.736047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:25.736672 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:25.736673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:25.736835 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:25.737416 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:25.737427 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:25.739360 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:25.741460 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:25.741466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:25.741469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:25.741816 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:25.742053 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:25.742054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:25.742054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:25.742207 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:25.742533 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:25.742539 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:25.742711 (+ 172us) client_negotiation.cc:769] Sending connection context
0621 05:23:25.743499 (+ 788us) client_negotiation.cc:240] Negotiation successful
0621 05:23:25.743659 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260621 05:23:25.744503 15313 heartbeater.cc:347] Connected to a master server at 127.2.32.62:43547
I20260621 05:23:25.744588 15313 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:25.744760 15313 heartbeater.cc:511] Master 127.2.32.62:43547 requested a full tablet report, sending...
W20260621 05:23:25.745378 15313 heartbeater.cc:644] Failed to heartbeat to 127.2.32.62:43547 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 05:23:25.745447 15313 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:25.745500 15313 heartbeater.cc:511] Master 127.2.32.62:43547 requested a full tablet report, sending...
I20260621 05:23:25.745836 15313 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:25.745905 15313 heartbeater.cc:511] Master 127.2.32.62:43547 requested a full tablet report, sending...
W20260621 05:23:25.746218 15313 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 05:23:25.831049 15316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:25.831439 15316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:25.831491 15316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:25.833534 15316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:25.833606 15316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:25.833657 15316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:23:25.835964 15316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.32.20:35677/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:35453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=33575
--tserver_master_addrs=127.2.32.62:43547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:25.836251 15316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:25.836521 15316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:25.842828 15324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:25.843007 15322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:25.843672 15316 server_base.cc:1034] running on GCE node
W20260621 05:23:25.843657 15321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:25.844141 15316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:25.844457 15316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:25.847791 15316 hybrid_clock.cc:648] HybridClock initialized: now 1782019405847766 us; error 47 us; skew 500 ppm
Jun 21 05:23:25 dist-test-slave-b357 krb5kdc[13985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019405, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:25.850550 15316 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:23:25.851084 15316 webserver.cc:460] Webserver started at http://127.2.32.3:33575/ using document root <none> and password file <none>
I20260621 05:23:25.851347 15316 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:25.851428 15316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 05:23:25 dist-test-slave-b357 krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019405, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
I20260621 05:23:25.864310 15316 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:23:25.865303 15332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:25.865473 15316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:23:25.865579 15316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "1c665e912d1040c2b8ad28e9200f8dbc"
format_stamp: "Formatted at 2026-06-21 05:23:25 on dist-test-slave-b357"
server_key: "528c5a10ea6bcfc54e0912a49aa38369"
server_key_iv: "6c184598c0f53809a34c09b85cc3f899"
server_key_version: "kuduclusterkey@0"
I20260621 05:23:25.865947 15316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:25.907078 15316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:25.910096 15316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:25.910282 15316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:25.910581 15316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:23:25.910974 15316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:23:25.911026 15316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:25.911072 15316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:23:25.911101 15316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:25.918725 15316 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:35453
I20260621 05:23:25.918882 15446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:35453 every 8 connection(s)
I20260621 05:23:25.919260 15316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:23:25.919822 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 15316
Jun 21 05:23:25 dist-test-slave-b357 krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019405, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:25.934610 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:25.923449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56824 (local address 127.2.32.62:43547)
0621 05:23:25.923527 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:25.923529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:25.923541 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:23:25.923573 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:25.923575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:25.923587 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:25.923654 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:25.926779 (+ 3125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:25.927630 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:25.928720 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:25.928924 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:25.929007 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:25.930478 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:25.930480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:25.930480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:25.930488 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:25.933323 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:25.933833 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:25.933834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:25.933835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:25.933893 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:25.934096 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:25.934097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:25.934097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:25.934215 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:25.934389 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:25.934425 (+ 36us) server_negotiation.cc:299] Negotiation successful
0621 05:23:25.934502 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":43}
I20260621 05:23:25.936353 15451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:25.923479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:53507 (local address 127.2.32.62:43547)
0621 05:23:25.925731 (+ 2252us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:25.925733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:25.925743 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:23:25.925765 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:25.925766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:25.925842 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:25.925939 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:25.926817 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:25.927650 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:25.929315 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:25.929636 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:25.929730 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:25.932452 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:25.932453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:25.932454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:25.932460 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:25.934319 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:25.935111 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:25.935112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:25.935113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:25.935262 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:25.935573 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:25.935574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:25.935574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:25.935731 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:25.935800 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:25.936151 (+ 351us) server_negotiation.cc:299] Negotiation successful
0621 05:23:25.936247 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":2216,"thread_start_us":54,"threads_started":1}
I20260621 05:23:25.936374 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:25.920300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:43547 (local address 127.2.32.3:53507)
0621 05:23:25.920487 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:25.920536 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:25.921191 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:25.926058 (+ 4867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:25.926063 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:25.926320 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:25.926634 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:25.926642 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:25.927784 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:25.927785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:25.929203 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:25.929206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:25.929765 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:25.929767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:25.929919 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:25.930484 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:25.930492 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:25.932124 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:25.934422 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:25.934429 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:25.934432 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:25.934957 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:25.935333 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:25.935334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:25.935334 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:25.935492 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:25.935823 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:25.935829 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:25.936007 (+ 178us) client_negotiation.cc:769] Sending connection context
0621 05:23:25.936096 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 05:23:25.936216 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":104,"threads_started":1}
I20260621 05:23:25.936808 15447 heartbeater.cc:347] Connected to a master server at 127.2.32.62:43547
I20260621 05:23:25.936901 15447 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:25.937073 15447 heartbeater.cc:511] Master 127.2.32.62:43547 requested a full tablet report, sending...
W20260621 05:23:25.937692 15447 heartbeater.cc:644] Failed to heartbeat to 127.2.32.62:43547 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 05:23:25.937749 15447 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:25.937809 15447 heartbeater.cc:511] Master 127.2.32.62:43547 requested a full tablet report, sending...
I20260621 05:23:25.938164 15447 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:25.938277 15447 heartbeater.cc:511] Master 127.2.32.62:43547 requested a full tablet report, sending...
W20260621 05:23:25.938604 15447 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 21 05:23:26 dist-test-slave-b357 krb5kdc[13985](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782019406, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:26.549634 15179 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:26.549798 15179 heartbeater.cc:511] Master 127.2.32.62:43547 requested a full tablet report, sending...
I20260621 05:23:26.550184 14980 ts_manager.cc:194] Registered new tserver with Master: 0382548e9bed473488d574fa766e9bf2 (127.2.32.1:40505)
I20260621 05:23:26.550874 14980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:46615
I20260621 05:23:26.749886 15313 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:26.750061 15313 heartbeater.cc:511] Master 127.2.32.62:43547 requested a full tablet report, sending...
I20260621 05:23:26.750406 14980 ts_manager.cc:194] Registered new tserver with Master: ee8536f60ee24a8f8495b917773101f3 (127.2.32.2:38179)
I20260621 05:23:26.750962 14980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:40189
Jun 21 05:23:26 dist-test-slave-b357 krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019406, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
I20260621 05:23:26.938808 15447 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:26.938984 15447 heartbeater.cc:511] Master 127.2.32.62:43547 requested a full tablet report, sending...
I20260621 05:23:26.939293 14980 ts_manager.cc:194] Registered new tserver with Master: 1c665e912d1040c2b8ad28e9200f8dbc (127.2.32.3:35453)
I20260621 05:23:26.939782 14980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:53507
I20260621 05:23:26.945150 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:23:26.954010 15468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:26.946097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56826 (local address 127.2.32.62:43547)
0621 05:23:26.946427 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:26.946429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:26.946520 (+ 91us) server_negotiation.cc:407] Connection header received
0621 05:23:26.946607 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:26.946608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:26.946618 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:26.946656 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:26.947395 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:26.948125 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:26.949099 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:26.949403 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:26.949455 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:26.951002 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:26.951004 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:26.951004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:26.951011 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:26.952767 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:26.953148 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:26.953149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:26.953149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:26.953206 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:26.953402 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:26.953402 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:26.953403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:26.953525 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:26.953596 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:26.953824 (+ 228us) server_negotiation.cc:299] Negotiation successful
0621 05:23:26.953907 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":65,"threads_started":1}
I20260621 05:23:26.956561 14980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:23:26.958998 14980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:23:26.968349 15475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:26.962169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:40505 (local address 127.0.0.1:57718)
0621 05:23:26.962946 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:26.962954 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:26.962986 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:26.963639 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:26.963642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:26.963645 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:23:26.963719 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:26.963721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:26.965307 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:26.965308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:26.966566 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:26.966569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:26.968035 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:26.968036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:26.968118 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:26.968131 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 05:23:26.968203 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 05:23:26.968260 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":61,"threads_started":1}
I20260621 05:23:26.968672 15478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:26.962265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57718 (local address 127.2.32.1:40505)
0621 05:23:26.963241 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:26.963246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:26.963258 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:23:26.963300 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:26.963302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:26.963317 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:26.963555 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:23:26.963805 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:26.965204 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:26.966658 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:26.967841 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:26.967913 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:26.967974 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:26.968553 (+ 579us) server_negotiation.cc:299] Negotiation successful
0621 05:23:26.968599 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":72,"threads_started":1}
I20260621 05:23:26.969347 15094 tablet_service.cc:1467] Processing CreateTablet for tablet d52bb02655a74e9cbe2ec3e6a66f9930 (DEFAULT_TABLE table=test-table [id=332fd3a499e546d1be51e1e6d99fcb3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:23:26.969861 15094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52bb02655a74e9cbe2ec3e6a66f9930. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:23:26.971212 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:26.962340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:35453 (local address 127.0.0.1:57742)
0621 05:23:26.963427 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:26.963436 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:26.963478 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:26.967194 (+ 3716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:26.967195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:26.967197 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:23:26.967238 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:26.967240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:26.968633 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:26.968634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:26.969908 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:26.969911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:26.970975 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:26.970976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:26.971034 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:26.971038 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:23:26.971082 (+ 44us) client_negotiation.cc:240] Negotiation successful
0621 05:23:26.971116 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"thread_start_us":69,"threads_started":1}
I20260621 05:23:26.971582 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:26.962438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57742 (local address 127.2.32.3:35453)
0621 05:23:26.966967 (+ 4529us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:26.966971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:26.966980 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:23:26.967011 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:26.967013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:26.967027 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:26.967090 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:23:26.967308 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:26.968336 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:26.970081 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:26.970818 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:26.970879 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:26.970930 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:26.971471 (+ 541us) server_negotiation.cc:299] Negotiation successful
0621 05:23:26.971513 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4497,"thread_start_us":64,"threads_started":1}
I20260621 05:23:26.972277 15381 tablet_service.cc:1467] Processing CreateTablet for tablet d52bb02655a74e9cbe2ec3e6a66f9930 (DEFAULT_TABLE table=test-table [id=332fd3a499e546d1be51e1e6d99fcb3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:23:26.972446 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:26.961941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:38179 (local address 127.0.0.1:56912)
0621 05:23:26.962495 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:26.962795 (+ 300us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:26.962850 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:26.966964 (+ 4114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:26.966965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:26.966966 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:23:26.967004 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:26.967006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:26.970147 (+ 3141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:26.970148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:26.971229 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:26.971232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:26.972238 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:26.972239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:26.972312 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:26.972316 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:23:26.972349 (+ 33us) client_negotiation.cc:240] Negotiation successful
0621 05:23:26.972383 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":87,"threads_started":1}
I20260621 05:23:26.972604 15381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52bb02655a74e9cbe2ec3e6a66f9930. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:23:26.973120 15476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:26.962044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56912 (local address 127.2.32.2:38179)
0621 05:23:26.966706 (+ 4662us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:26.966710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:26.966728 (+ 18us) server_negotiation.cc:407] Connection header received
0621 05:23:26.966759 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:26.966761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:26.966776 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:26.966852 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:23:26.968784 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:26.970050 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:26.971332 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:26.972135 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:26.972610 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:26.972669 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:26.972708 (+ 39us) server_negotiation.cc:299] Negotiation successful
0621 05:23:26.972749 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4630,"thread_start_us":50,"threads_started":1}
I20260621 05:23:26.973603 15247 tablet_service.cc:1467] Processing CreateTablet for tablet d52bb02655a74e9cbe2ec3e6a66f9930 (DEFAULT_TABLE table=test-table [id=332fd3a499e546d1be51e1e6d99fcb3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:23:26.973882 15247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52bb02655a74e9cbe2ec3e6a66f9930. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:23:26.976346 15482 tablet_bootstrap.cc:492] T d52bb02655a74e9cbe2ec3e6a66f9930 P 1c665e912d1040c2b8ad28e9200f8dbc: Bootstrap starting.
I20260621 05:23:26.976939 15481 tablet_bootstrap.cc:492] T d52bb02655a74e9cbe2ec3e6a66f9930 P 0382548e9bed473488d574fa766e9bf2: Bootstrap starting.
I20260621 05:23:26.976971 15482 tablet_bootstrap.cc:654] T d52bb02655a74e9cbe2ec3e6a66f9930 P 1c665e912d1040c2b8ad28e9200f8dbc: Neither blocks nor log segments found. Creating new log.
I20260621 05:23:26.977277 15482 log.cc:824] T d52bb02655a74e9cbe2ec3e6a66f9930 P 1c665e912d1040c2b8ad28e9200f8dbc: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:26.977545 15481 tablet_bootstrap.cc:654] T d52bb02655a74e9cbe2ec3e6a66f9930 P 0382548e9bed473488d574fa766e9bf2: Neither blocks nor log segments found. Creating new log.
I20260621 05:23:26.977589 15483 tablet_bootstrap.cc:492] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3: Bootstrap starting.
I20260621 05:23:26.978271 15481 log.cc:824] T d52bb02655a74e9cbe2ec3e6a66f9930 P 0382548e9bed473488d574fa766e9bf2: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:26.978691 15483 tablet_bootstrap.cc:654] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3: Neither blocks nor log segments found. Creating new log.
I20260621 05:23:26.979068 15483 log.cc:824] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:26.979971 15482 tablet_bootstrap.cc:492] T d52bb02655a74e9cbe2ec3e6a66f9930 P 1c665e912d1040c2b8ad28e9200f8dbc: No bootstrap required, opened a new log
I20260621 05:23:26.980054 15482 ts_tablet_manager.cc:1397] T d52bb02655a74e9cbe2ec3e6a66f9930 P 1c665e912d1040c2b8ad28e9200f8dbc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260621 05:23:26.980120 15481 tablet_bootstrap.cc:492] T d52bb02655a74e9cbe2ec3e6a66f9930 P 0382548e9bed473488d574fa766e9bf2: No bootstrap required, opened a new log
I20260621 05:23:26.980183 15481 ts_tablet_manager.cc:1397] T d52bb02655a74e9cbe2ec3e6a66f9930 P 0382548e9bed473488d574fa766e9bf2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:23:26.982851 15481 raft_consensus.cc:348] T d52bb02655a74e9cbe2ec3e6a66f9930 P 0382548e9bed473488d574fa766e9bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8536f60ee24a8f8495b917773101f3" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38179 } } peers { permanent_uuid: "1c665e912d1040c2b8ad28e9200f8dbc" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 35453 } } peers { permanent_uuid: "0382548e9bed473488d574fa766e9bf2" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40505 } }
I20260621 05:23:26.982894 15482 raft_consensus.cc:348] T d52bb02655a74e9cbe2ec3e6a66f9930 P 1c665e912d1040c2b8ad28e9200f8dbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8536f60ee24a8f8495b917773101f3" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38179 } } peers { permanent_uuid: "1c665e912d1040c2b8ad28e9200f8dbc" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 35453 } } peers { permanent_uuid: "0382548e9bed473488d574fa766e9bf2" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40505 } }
I20260621 05:23:26.983028 15481 raft_consensus.cc:374] T d52bb02655a74e9cbe2ec3e6a66f9930 P 0382548e9bed473488d574fa766e9bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:23:26.983062 15481 raft_consensus.cc:729] T d52bb02655a74e9cbe2ec3e6a66f9930 P 0382548e9bed473488d574fa766e9bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0382548e9bed473488d574fa766e9bf2, State: Initialized, Role: FOLLOWER
I20260621 05:23:26.983057 15482 raft_consensus.cc:374] T d52bb02655a74e9cbe2ec3e6a66f9930 P 1c665e912d1040c2b8ad28e9200f8dbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:23:26.983100 15482 raft_consensus.cc:729] T d52bb02655a74e9cbe2ec3e6a66f9930 P 1c665e912d1040c2b8ad28e9200f8dbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c665e912d1040c2b8ad28e9200f8dbc, State: Initialized, Role: FOLLOWER
I20260621 05:23:26.983278 15482 consensus_queue.cc:260] T d52bb02655a74e9cbe2ec3e6a66f9930 P 1c665e912d1040c2b8ad28e9200f8dbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8536f60ee24a8f8495b917773101f3" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38179 } } peers { permanent_uuid: "1c665e912d1040c2b8ad28e9200f8dbc" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 35453 } } peers { permanent_uuid: "0382548e9bed473488d574fa766e9bf2" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40505 } }
I20260621 05:23:26.983551 15482 ts_tablet_manager.cc:1428] T d52bb02655a74e9cbe2ec3e6a66f9930 P 1c665e912d1040c2b8ad28e9200f8dbc: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260621 05:23:26.983530 15481 consensus_queue.cc:260] T d52bb02655a74e9cbe2ec3e6a66f9930 P 0382548e9bed473488d574fa766e9bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8536f60ee24a8f8495b917773101f3" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38179 } } peers { permanent_uuid: "1c665e912d1040c2b8ad28e9200f8dbc" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 35453 } } peers { permanent_uuid: "0382548e9bed473488d574fa766e9bf2" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40505 } }
I20260621 05:23:26.983682 15447 heartbeater.cc:503] Master 127.2.32.62:43547 was elected leader, sending a full tablet report...
I20260621 05:23:26.983946 15481 ts_tablet_manager.cc:1428] T d52bb02655a74e9cbe2ec3e6a66f9930 P 0382548e9bed473488d574fa766e9bf2: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260621 05:23:26.984128 15179 heartbeater.cc:503] Master 127.2.32.62:43547 was elected leader, sending a full tablet report...
I20260621 05:23:26.987689 15483 tablet_bootstrap.cc:492] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3: No bootstrap required, opened a new log
I20260621 05:23:26.987771 15483 ts_tablet_manager.cc:1397] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260621 05:23:26.990113 15483 raft_consensus.cc:348] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8536f60ee24a8f8495b917773101f3" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38179 } } peers { permanent_uuid: "1c665e912d1040c2b8ad28e9200f8dbc" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 35453 } } peers { permanent_uuid: "0382548e9bed473488d574fa766e9bf2" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40505 } }
I20260621 05:23:26.990438 15483 raft_consensus.cc:374] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:23:26.990520 15483 raft_consensus.cc:729] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee8536f60ee24a8f8495b917773101f3, State: Initialized, Role: FOLLOWER
I20260621 05:23:26.990684 15483 consensus_queue.cc:260] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8536f60ee24a8f8495b917773101f3" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38179 } } peers { permanent_uuid: "1c665e912d1040c2b8ad28e9200f8dbc" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 35453 } } peers { permanent_uuid: "0382548e9bed473488d574fa766e9bf2" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40505 } }
I20260621 05:23:26.991072 15313 heartbeater.cc:503] Master 127.2.32.62:43547 was elected leader, sending a full tablet report...
I20260621 05:23:26.992332 15483 ts_tablet_manager.cc:1428] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
W20260621 05:23:27.029017 15180 tablet.cc:2369] T d52bb02655a74e9cbe2ec3e6a66f9930 P 0382548e9bed473488d574fa766e9bf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:23:27.103479 15489 raft_consensus.cc:482] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:23:27.103657 15489 raft_consensus.cc:504] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8536f60ee24a8f8495b917773101f3" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38179 } } peers { permanent_uuid: "1c665e912d1040c2b8ad28e9200f8dbc" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 35453 } } peers { permanent_uuid: "0382548e9bed473488d574fa766e9bf2" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40505 } }
I20260621 05:23:27.104144 15489 leader_election.cc:288] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c665e912d1040c2b8ad28e9200f8dbc (127.2.32.3:35453), 0382548e9bed473488d574fa766e9bf2 (127.2.32.1:40505)
I20260621 05:23:27.108635 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:27.104885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:38581 (local address 127.2.32.3:35453)
0621 05:23:27.104936 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:27.104938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:27.104946 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:23:27.104973 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:27.104974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:27.104985 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:27.105030 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:23:27.105251 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:27.105999 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:27.107603 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:27.108192 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:27.108243 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:27.108255 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:27.108522 (+ 267us) server_negotiation.cc:299] Negotiation successful
0621 05:23:27.108566 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260621 05:23:27.109277 15490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:27.104424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:35453 (local address 127.2.32.2:38581)
0621 05:23:27.104681 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:27.104693 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:27.104763 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:27.105078 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:27.105079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:27.105082 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:23:27.105160 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:27.105163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:27.106114 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:27.106115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:27.107489 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:27.107492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:27.108356 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:27.108357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:27.108436 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:27.108439 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:23:27.109126 (+ 687us) client_negotiation.cc:240] Negotiation successful
0621 05:23:27.109177 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":49,"thread_start_us":110,"threads_started":1}
I20260621 05:23:27.109279 15478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:27.104778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:40991 (local address 127.2.32.1:40505)
0621 05:23:27.104840 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:27.104843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:27.105062 (+ 219us) server_negotiation.cc:407] Connection header received
0621 05:23:27.105173 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:27.105175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:27.105189 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:27.105217 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:23:27.105461 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:27.106197 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:27.107803 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:27.108385 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:27.108431 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:27.108458 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:27.109158 (+ 700us) server_negotiation.cc:299] Negotiation successful
0621 05:23:27.109213 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260621 05:23:27.109608 15401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52bb02655a74e9cbe2ec3e6a66f9930" candidate_uuid: "ee8536f60ee24a8f8495b917773101f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c665e912d1040c2b8ad28e9200f8dbc" is_pre_election: true
I20260621 05:23:27.109728 15491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:27.104459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:40505 (local address 127.2.32.2:40991)
0621 05:23:27.104987 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:27.104998 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:27.105094 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:27.105289 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:27.105290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:27.105292 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:23:27.105393 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:27.105396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:27.106302 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:27.106303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:27.107627 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:27.107631 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:27.108598 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:27.108599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:27.108676 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:27.108679 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:23:27.109610 (+ 931us) client_negotiation.cc:240] Negotiation successful
0621 05:23:27.109656 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":178,"threads_started":1}
I20260621 05:23:27.109779 15401 raft_consensus.cc:2454] T d52bb02655a74e9cbe2ec3e6a66f9930 P 1c665e912d1040c2b8ad28e9200f8dbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee8536f60ee24a8f8495b917773101f3 in term 0.
I20260621 05:23:27.110046 15203 leader_election.cc:302] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c665e912d1040c2b8ad28e9200f8dbc, ee8536f60ee24a8f8495b917773101f3; no voters:
I20260621 05:23:27.110057 15116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52bb02655a74e9cbe2ec3e6a66f9930" candidate_uuid: "ee8536f60ee24a8f8495b917773101f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0382548e9bed473488d574fa766e9bf2" is_pre_election: true
I20260621 05:23:27.110205 15116 raft_consensus.cc:2454] T d52bb02655a74e9cbe2ec3e6a66f9930 P 0382548e9bed473488d574fa766e9bf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee8536f60ee24a8f8495b917773101f3 in term 0.
I20260621 05:23:27.110183 15489 raft_consensus.cc:2781] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:23:27.110366 15489 raft_consensus.cc:482] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:23:27.110410 15489 raft_consensus.cc:3037] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:23:27.111392 15489 raft_consensus.cc:504] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8536f60ee24a8f8495b917773101f3" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38179 } } peers { permanent_uuid: "1c665e912d1040c2b8ad28e9200f8dbc" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 35453 } } peers { permanent_uuid: "0382548e9bed473488d574fa766e9bf2" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40505 } }
I20260621 05:23:27.111531 15489 leader_election.cc:288] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [CANDIDATE]: Term 1 election: Requested vote from peers 1c665e912d1040c2b8ad28e9200f8dbc (127.2.32.3:35453), 0382548e9bed473488d574fa766e9bf2 (127.2.32.1:40505)
I20260621 05:23:27.111711 15116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52bb02655a74e9cbe2ec3e6a66f9930" candidate_uuid: "ee8536f60ee24a8f8495b917773101f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0382548e9bed473488d574fa766e9bf2"
I20260621 05:23:27.111753 15401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52bb02655a74e9cbe2ec3e6a66f9930" candidate_uuid: "ee8536f60ee24a8f8495b917773101f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c665e912d1040c2b8ad28e9200f8dbc"
I20260621 05:23:27.111776 15116 raft_consensus.cc:3037] T d52bb02655a74e9cbe2ec3e6a66f9930 P 0382548e9bed473488d574fa766e9bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:23:27.111825 15401 raft_consensus.cc:3037] T d52bb02655a74e9cbe2ec3e6a66f9930 P 1c665e912d1040c2b8ad28e9200f8dbc [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:23:27.112564 15116 raft_consensus.cc:2454] T d52bb02655a74e9cbe2ec3e6a66f9930 P 0382548e9bed473488d574fa766e9bf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee8536f60ee24a8f8495b917773101f3 in term 1.
I20260621 05:23:27.112748 15401 raft_consensus.cc:2454] T d52bb02655a74e9cbe2ec3e6a66f9930 P 1c665e912d1040c2b8ad28e9200f8dbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee8536f60ee24a8f8495b917773101f3 in term 1.
I20260621 05:23:27.112792 15201 leader_election.cc:302] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0382548e9bed473488d574fa766e9bf2, ee8536f60ee24a8f8495b917773101f3; no voters:
I20260621 05:23:27.112927 15489 raft_consensus.cc:2781] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:23:27.113081 15489 raft_consensus.cc:686] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [term 1 LEADER]: Becoming Leader. State: Replica: ee8536f60ee24a8f8495b917773101f3, State: Running, Role: LEADER
I20260621 05:23:27.113169 15489 consensus_queue.cc:237] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8536f60ee24a8f8495b917773101f3" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38179 } } peers { permanent_uuid: "1c665e912d1040c2b8ad28e9200f8dbc" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 35453 } } peers { permanent_uuid: "0382548e9bed473488d574fa766e9bf2" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40505 } }
I20260621 05:23:27.113904 14980 catalog_manager.cc:5495] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee8536f60ee24a8f8495b917773101f3 (127.2.32.2). New cstate: current_term: 1 leader_uuid: "ee8536f60ee24a8f8495b917773101f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8536f60ee24a8f8495b917773101f3" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 38179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c665e912d1040c2b8ad28e9200f8dbc" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 35453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0382548e9bed473488d574fa766e9bf2" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40505 } health_report { overall_health: UNKNOWN } } }
I20260621 05:23:27.151284 15476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:27.147920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.2.32.2:38179)
0621 05:23:27.147966 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:27.147968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:27.147975 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:23:27.148017 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:27.148018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:27.148042 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:27.148108 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:23:27.148322 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:27.149023 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:27.150142 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:27.150420 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:27.150484 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:27.150694 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:23:27.150823 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:23:27.150902 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:27.151171 (+ 269us) server_negotiation.cc:299] Negotiation successful
0621 05:23:27.151193 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:23:27.153370 15116 raft_consensus.cc:1261] T d52bb02655a74e9cbe2ec3e6a66f9930 P 0382548e9bed473488d574fa766e9bf2 [term 1 FOLLOWER]: Refusing update from remote peer ee8536f60ee24a8f8495b917773101f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:23:27.153426 15401 raft_consensus.cc:1261] T d52bb02655a74e9cbe2ec3e6a66f9930 P 1c665e912d1040c2b8ad28e9200f8dbc [term 1 FOLLOWER]: Refusing update from remote peer ee8536f60ee24a8f8495b917773101f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:23:27.153703 15489 consensus_queue.cc:1035] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0382548e9bed473488d574fa766e9bf2" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 40505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:23:27.153829 15492 consensus_queue.cc:1035] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c665e912d1040c2b8ad28e9200f8dbc" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 35453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:23:27.156754 15497 mvcc.cc:205] Tried to move back new op lower bound from 7299151491697532928 to 7299151491536060416. Current Snapshot: MvccSnapshot[applied={T|T < 7299151491697532928}]
I20260621 05:23:27.157297 15496 mvcc.cc:205] Tried to move back new op lower bound from 7299151491697532928 to 7299151491536060416. Current Snapshot: MvccSnapshot[applied={T|T < 7299151491697532928 or (T in {7299151491697532928})}]
I20260621 05:23:27.159335 14980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:23:27.159428 14980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:23:27.161171 14980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f10936fcc2049c1aae31332d92af20b: Sending DeleteTablet for 3 replicas of tablet d52bb02655a74e9cbe2ec3e6a66f9930
I20260621 05:23:27.161736 15247 tablet_service.cc:1514] Processing DeleteTablet for tablet d52bb02655a74e9cbe2ec3e6a66f9930 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:23:27 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:56912
I20260621 05:23:27.161919 15094 tablet_service.cc:1514] Processing DeleteTablet for tablet d52bb02655a74e9cbe2ec3e6a66f9930 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:23:27 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:57718
I20260621 05:23:27.162094 15381 tablet_service.cc:1514] Processing DeleteTablet for tablet d52bb02655a74e9cbe2ec3e6a66f9930 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:23:27 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:57742
I20260621 05:23:27.163885 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 15019
W20260621 05:23:27.170917 15448 tablet.cc:2369] T d52bb02655a74e9cbe2ec3e6a66f9930 P 1c665e912d1040c2b8ad28e9200f8dbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:23:27.172127 15499 mvcc.cc:205] Tried to move back new op lower bound from 7299151491697532928 to 7299151491536060416. Current Snapshot: MvccSnapshot[applied={T|T < 7299151491697532928}]
I20260621 05:23:27.172397 15504 tablet_replica.cc:330] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3: stopping tablet replica
Jun 21 05:23:27 dist-test-slave-b357 krb5kdc[13985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019406, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for HTTP/127.2.32.20@KRBTEST.COM
I20260621 05:23:27.174923 15506 tablet_replica.cc:330] T d52bb02655a74e9cbe2ec3e6a66f9930 P 1c665e912d1040c2b8ad28e9200f8dbc: stopping tablet replica
I20260621 05:23:27.175098 15504 raft_consensus.cc:2229] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:23:27.175233 15504 raft_consensus.cc:2258] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:23:27.175357 15506 raft_consensus.cc:2229] T d52bb02655a74e9cbe2ec3e6a66f9930 P 1c665e912d1040c2b8ad28e9200f8dbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:23:27.175420 15506 raft_consensus.cc:2258] T d52bb02655a74e9cbe2ec3e6a66f9930 P 1c665e912d1040c2b8ad28e9200f8dbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:23:27.175923 15504 ts_tablet_manager.cc:1905] T d52bb02655a74e9cbe2ec3e6a66f9930 P ee8536f60ee24a8f8495b917773101f3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260621 05:23:27.177171 14966 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.32.1:40505 (error 108)
I20260621 05:23:27.177811 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 15183
I20260621 05:23:27.183887 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:27.183706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:40505 (local address 127.0.0.1:57724)
0621 05:23:27.183782 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:27.183837 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.1:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260621 05:23:27.184144 14966 catalog_manager.cc:4584] TS 0382548e9bed473488d574fa766e9bf2 (127.2.32.1:40505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d52bb02655a74e9cbe2ec3e6a66f9930: Network error: Client connection negotiation failed: client connection to 127.2.32.1:40505: connect: Connection refused (error 111)
I20260621 05:23:27.185283 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 15316
I20260621 05:23:27.185403 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:27.185290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:38179 (local address 127.0.0.1:56940)
0621 05:23:27.185342 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:27.185366 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.2:38179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260621 05:23:27.194412 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 14948
I20260621 05:23:27.207370 2176 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260621 05:23:27.552624 2176 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260621 05:23:27.552731 2176 mini_ranger.cc:66] Stopping Ranger...
I20260621 05:23:27.898677 2176 mini_ranger.cc:68] Stopped Ranger
2026-06-21 05:23:27.899 UTC [14015] LOG: received smart shutdown request
2026-06-21 05:23:27.903 UTC [14015] LOG: background worker "logical replication launcher" (PID 14024) exited with exit code 1
2026-06-21 05:23:27.903 UTC [14019] LOG: shutting down
2026-06-21 05:23:28.108 UTC [14015] LOG: database system is shut down
2026-06-21T05:23:28Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50630 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:23:28 dist-test-slave-b357 krb5kdc[15518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:23:28 dist-test-slave-b357 krb5kdc[15518](info): set up 2 sockets
Jun 21 05:23:28 dist-test-slave-b357 krb5kdc[15518](info): commencing operation
krb5kdc: starting...
W20260621 05:23:29.291237 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:23:29 dist-test-slave-b357 krb5kdc[15518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:23:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:23:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:23:29.419247 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:34291
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:38395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:34291,127.2.32.61:37545,127.2.32.60:35387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:23:29.504896 15531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:29.505103 15531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:29.505125 15531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:29.506237 15531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:23:29.506278 15531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:29.506294 15531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:29.506305 15531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:23:29.506317 15531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:23:29.507910 15531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:38395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:34291,127.2.32.61:37545,127.2.32.60:35387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:34291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:29.508137 15531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:29.508385 15531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:29.513595 15536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:29.513798 15539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:29.513897 15531 server_base.cc:1034] running on GCE node
W20260621 05:23:29.513633 15537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:29.514401 15531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:29.514760 15531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:29.515946 15531 hybrid_clock.cc:648] HybridClock initialized: now 1782019409515931 us; error 46 us; skew 500 ppm
Jun 21 05:23:29 dist-test-slave-b357 krb5kdc[15518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019409, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:29.518515 15531 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:23:29.519028 15531 webserver.cc:460] Webserver started at http://127.2.32.62:42541/ using document root <none> and password file <none>
I20260621 05:23:29.519279 15531 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:29.519346 15531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:29.519481 15531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:23:29.520507 15531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "ff960a5759954f1aa7dac71c333dd266"
format_stamp: "Formatted at 2026-06-21 05:23:29 on dist-test-slave-b357"
server_key: "260ab51ad1cd25d6ef22828dbabc1ca3"
server_key_iv: "fc6001c4652954abeb334bb6fb7fe71e"
server_key_version: "encryptionkey@0"
I20260621 05:23:29.520872 15531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "ff960a5759954f1aa7dac71c333dd266"
format_stamp: "Formatted at 2026-06-21 05:23:29 on dist-test-slave-b357"
server_key: "260ab51ad1cd25d6ef22828dbabc1ca3"
server_key_iv: "fc6001c4652954abeb334bb6fb7fe71e"
server_key_version: "encryptionkey@0"
I20260621 05:23:29.522290 15531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:23:29.523183 15546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:29.523497 15531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:29.523628 15531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "ff960a5759954f1aa7dac71c333dd266"
format_stamp: "Formatted at 2026-06-21 05:23:29 on dist-test-slave-b357"
server_key: "260ab51ad1cd25d6ef22828dbabc1ca3"
server_key_iv: "fc6001c4652954abeb334bb6fb7fe71e"
server_key_version: "encryptionkey@0"
I20260621 05:23:29.523742 15531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:29.548316 15531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:29.550936 15531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:29.551163 15531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:29.555962 15531 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:34291
I20260621 05:23:29.556026 15598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:34291 every 8 connection(s)
I20260621 05:23:29.556537 15531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:23:29.557245 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:23:29.558728 15599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:23:29.565717 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 15531
I20260621 05:23:29.565861 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:23:29.566084 2176 external_mini_cluster.cc:1428] Setting key 0c209f30fbe70ffcc508a8a790963689
Jun 21 05:23:29 dist-test-slave-b357 krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019409, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:29.570920 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.559598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32904 (local address 127.2.32.62:34291)
0621 05:23:29.560767 (+ 1169us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:29.560771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:29.560783 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:23:29.560859 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:29.560860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:29.560943 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:29.561054 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:29.561751 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.562718 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.563914 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.564317 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.564418 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.567066 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:29.567076 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:29.567077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:29.567088 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:29.569053 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.569598 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.569603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.569604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.569702 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.570030 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.570031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.570032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.570279 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:29.570366 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:29.570642 (+ 276us) server_negotiation.cc:299] Negotiation successful
0621 05:23:29.570753 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1130,"thread_start_us":133,"threads_started":1}
I20260621 05:23:29.571035 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.559598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:34291 (local address 127.0.0.1:32904)
0621 05:23:29.559913 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.559963 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:29.560606 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:29.561104 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:29.561106 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:29.561443 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:29.561629 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:29.561633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.562830 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.562831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:29.563793 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:29.563796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.564425 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.564426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:29.564538 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.565158 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:29.565168 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:29.566943 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:29.569157 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:29.569164 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:29.569166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:29.569477 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:29.569788 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:29.569789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:29.569790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:29.569908 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:29.570370 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:29.570378 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:29.570467 (+ 89us) client_negotiation.cc:769] Sending connection context
0621 05:23:29.570585 (+ 118us) client_negotiation.cc:240] Negotiation successful
0621 05:23:29.570843 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":65,"thread_start_us":131,"threads_started":1}
I20260621 05:23:29.571851 15599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:23:29.572232 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.572092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:37545 (local address 127.0.0.1:42194)
0621 05:23:29.572147 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.572205 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.61:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260621 05:23:29.572373 15547 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.32.61:37545: connect: Connection refused (error 111)
I20260621 05:23:29.572839 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.572731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:37545 (local address 127.0.0.1:42196)
0621 05:23:29.572790 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.572819 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.61:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260621 05:23:29.572988 15599 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.61:37545: Network error: Client connection negotiation failed: client connection to 127.2.32.61:37545: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.2.32.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.61.keytab.
Entry for principal kudu/127.2.32.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.61.keytab.
I20260621 05:23:29.594267 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.61:37545
--webserver_interface=127.2.32.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.61.keytab
--principal=kudu/127.2.32.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:38395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:34291,127.2.32.61:37545,127.2.32.60:35387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:23:29.622275 15599 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } attempt: 1
I20260621 05:23:29.622901 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.622746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:37545 (local address 127.0.0.1:42212)
0621 05:23:29.622820 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.622867 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.61:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260621 05:23:29.623306 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.623204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:37545 (local address 127.0.0.1:42220)
0621 05:23:29.623255 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.623281 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.61:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260621 05:23:29.623474 15599 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.61:37545: Network error: Client connection negotiation failed: client connection to 127.2.32.61:37545: connect: Connection refused (error 111)
W20260621 05:23:29.678318 15606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:29.678511 15606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:29.678534 15606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:29.679690 15606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:23:29.679733 15606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:29.679749 15606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:29.679760 15606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:23:29.679773 15606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:23:29.681190 15606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:38395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:34291,127.2.32.61:37545,127.2.32.60:35387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.61.keytab
--principal=kudu/127.2.32.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.61:37545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.32.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:29.681370 15606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:29.681528 15606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:29.686295 15612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:29.686264 15614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:29.686452 15611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:29.686581 15606 server_base.cc:1034] running on GCE node
I20260621 05:23:29.686910 15606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:29.687251 15606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:29.688406 15606 hybrid_clock.cc:648] HybridClock initialized: now 1782019409688398 us; error 53 us; skew 500 ppm
Jun 21 05:23:29 dist-test-slave-b357 krb5kdc[15518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019409, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:29.690966 15606 init.cc:377] Logged in from keytab as kudu/127.2.32.61@KRBTEST.COM (short username kudu)
I20260621 05:23:29.691432 15606 webserver.cc:460] Webserver started at http://127.2.32.61:44637/ using document root <none> and password file <none>
I20260621 05:23:29.691668 15606 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:29.691733 15606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:29.691690 15599 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } attempt: 2
I20260621 05:23:29.691851 15606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:23:29.692109 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.691985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:37545 (local address 127.0.0.1:42226)
0621 05:23:29.692043 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.692080 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.61:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260621 05:23:29.692410 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.692324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:37545 (local address 127.0.0.1:42234)
0621 05:23:29.692365 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.692389 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.61:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260621 05:23:29.692510 15599 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.61:37545: Network error: Client connection negotiation failed: client connection to 127.2.32.61:37545: connect: Connection refused (error 111)
I20260621 05:23:29.693086 15606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-1/data/instance:
uuid: "d51128e98e7645309c8009da172f6c39"
format_stamp: "Formatted at 2026-06-21 05:23:29 on dist-test-slave-b357"
server_key: "752c81afc3f1ad64b4f0d2681d364853"
server_key_iv: "21b75ee876040cf82b0468d86c3c8a7a"
server_key_version: "encryptionkey@0"
I20260621 05:23:29.693444 15606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-1/wal/instance:
uuid: "d51128e98e7645309c8009da172f6c39"
format_stamp: "Formatted at 2026-06-21 05:23:29 on dist-test-slave-b357"
server_key: "752c81afc3f1ad64b4f0d2681d364853"
server_key_iv: "21b75ee876040cf82b0468d86c3c8a7a"
server_key_version: "encryptionkey@0"
I20260621 05:23:29.694796 15606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:23:29.695660 15621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:29.695981 15606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:23:29.696166 15606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-1/wal
uuid: "d51128e98e7645309c8009da172f6c39"
format_stamp: "Formatted at 2026-06-21 05:23:29 on dist-test-slave-b357"
server_key: "752c81afc3f1ad64b4f0d2681d364853"
server_key_iv: "21b75ee876040cf82b0468d86c3c8a7a"
server_key_version: "encryptionkey@0"
I20260621 05:23:29.696292 15606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:29.708211 15606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:29.710623 15606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:29.710788 15606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:29.715391 15606 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.61:37545
I20260621 05:23:29.715427 15673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.61:37545 every 8 connection(s)
I20260621 05:23:29.715775 15606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-1/data/info.pb
I20260621 05:23:29.716614 15674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:23:29.718137 15674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:23:29.720036 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 15606
I20260621 05:23:29.720166 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-1/wal/instance
I20260621 05:23:29.720443 2176 external_mini_cluster.cc:1428] Setting key 5f06ab85e9db874e9edaf842371c6279
Jun 21 05:23:29 dist-test-slave-b357 krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019409, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.61@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:29.729856 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.718846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32910 (local address 127.2.32.62:34291)
0621 05:23:29.718912 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:29.718914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:29.719198 (+ 284us) server_negotiation.cc:407] Connection header received
0621 05:23:29.719680 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:29.719682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:29.719755 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:29.719871 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:29.720668 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.721595 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.722915 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.723346 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.723455 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.726377 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:29.726379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:29.726380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:29.726390 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:29.728249 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.728716 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.728718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.728718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.728785 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.729044 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.729045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.729045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.729180 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:29.729253 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:29.729655 (+ 402us) server_negotiation.cc:299] Negotiation successful
0621 05:23:29.729757 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 05:23:29.730067 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.718815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:34291 (local address 127.0.0.1:32910)
0621 05:23:29.719004 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.719038 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:29.719541 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:29.719952 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:29.719956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:29.720312 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:29.720550 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:29.720561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.721773 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.721774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:29.722804 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:29.722807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.723515 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.723516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:29.723713 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.724346 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:29.724359 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:29.726252 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:29.728347 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:29.728353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:29.728356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:29.728618 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:29.728860 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:29.728861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:29.728861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:29.728974 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:29.729265 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:29.729271 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:29.729408 (+ 137us) client_negotiation.cc:769] Sending connection context
0621 05:23:29.729630 (+ 222us) client_negotiation.cc:240] Negotiation successful
0621 05:23:29.729831 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.2.32.60@KRBTEST.COM; defaulting to no policy
I20260621 05:23:29.730789 15674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.60@KRBTEST.COM" created.
Jun 21 05:23:29 dist-test-slave-b357 krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019409, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.61@KRBTEST.COM for kudu/127.2.32.61@KRBTEST.COM
I20260621 05:23:29.739754 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.731050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:37545 (local address 127.0.0.1:42246)
0621 05:23:29.731111 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.731121 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:29.731177 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:29.731517 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:29.731518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:29.731673 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:29.731736 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:29.731738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.732761 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.732762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:29.733272 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:29.733274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.734035 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.734036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:29.734123 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.734549 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:29.734555 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:29.736281 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:29.738452 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:29.738453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:29.738454 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:29.738644 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:29.738884 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:29.738885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:29.738885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:29.738946 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:29.739341 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:29.739342 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:29.739384 (+ 42us) client_negotiation.cc:769] Sending connection context
0621 05:23:29.739478 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 05:23:29.739624 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260621 05:23:29.739789 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.731200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42246 (local address 127.2.32.61:37545)
0621 05:23:29.731367 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:29.731370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:29.731386 (+ 16us) server_negotiation.cc:407] Connection header received
0621 05:23:29.731405 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:29.731406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:29.731417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:29.731483 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:29.731848 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.732535 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.733404 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.733870 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.733955 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.736428 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:29.736448 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:29.736448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:29.736456 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:29.738330 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.738747 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.738750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.738750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.738817 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.739023 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.739023 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.739024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.739263 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:29.739355 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:29.739515 (+ 160us) server_negotiation.cc:299] Negotiation successful
0621 05:23:29.739684 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":136,"thread_start_us":108,"threads_started":1}
I20260621 05:23:29.740748 15674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:23:29.741175 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.741017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:35387 (local address 127.0.0.1:55578)
0621 05:23:29.741078 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.741136 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:35387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260621 05:23:29.741413 15625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.32.60:35387: connect: Connection refused (error 111)
I20260621 05:23:29.741978 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.741851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:35387 (local address 127.0.0.1:55584)
0621 05:23:29.741920 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.741958 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:35387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260621 05:23:29.742136 15674 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.60:35387: Network error: Client connection negotiation failed: client connection to 127.2.32.60:35387: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.60.keytab.
Entry for principal kudu/127.2.32.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.60.keytab.
I20260621 05:23:29.750488 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.60:35387
--webserver_interface=127.2.32.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.60.keytab
--principal=kudu/127.2.32.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:38395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:34291,127.2.32.61:37545,127.2.32.60:35387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:23:29.783656 15599 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } attempt: 3
Jun 21 05:23:29 dist-test-slave-b357 krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019409, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for kudu/127.2.32.61@KRBTEST.COM
I20260621 05:23:29.791358 15674 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } attempt: 1
I20260621 05:23:29.791781 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.791656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:35387 (local address 127.0.0.1:55594)
0621 05:23:29.791715 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.791749 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:35387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260621 05:23:29.792331 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.784113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42258 (local address 127.2.32.61:37545)
0621 05:23:29.784198 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:29.784200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:29.784226 (+ 26us) server_negotiation.cc:407] Connection header received
0621 05:23:29.784254 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:29.784255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:29.784268 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:29.784320 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:29.784638 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.785444 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.786212 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.786570 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.786661 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.789224 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:29.789225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:29.789226 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:29.789233 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:29.790972 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.791450 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.791451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.791452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.791521 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.791753 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.791754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.791754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.791886 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:29.791953 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:29.792160 (+ 207us) server_negotiation.cc:299] Negotiation successful
0621 05:23:29.792245 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 05:23:29.792335 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.784014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:37545 (local address 127.0.0.1:42258)
0621 05:23:29.784084 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.784098 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:29.784181 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:29.784334 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:29.784334 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:29.784476 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:29.784560 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:29.784562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.785581 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.785582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:29.786102 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:29.786103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.786671 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.786672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:29.786751 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.787367 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:29.787388 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:29.789090 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:29.791079 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:29.791080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:29.791080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:29.791347 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:29.791586 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:29.791592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:29.791592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:29.791669 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:29.791982 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:29.791983 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:29.792036 (+ 53us) client_negotiation.cc:769] Sending connection context
0621 05:23:29.792124 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 05:23:29.792235 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260621 05:23:29.792572 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.792484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:35387 (local address 127.0.0.1:55600)
0621 05:23:29.792517 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.792548 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:35387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260621 05:23:29.792645 15674 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.60:35387: Network error: Client connection negotiation failed: client connection to 127.2.32.60:35387: connect: Connection refused (error 111)
I20260621 05:23:29.792734 15599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:23:29.793054 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.792946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:35387 (local address 127.0.0.1:55608)
0621 05:23:29.793001 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.793031 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:35387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260621 05:23:29.793336 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.793261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:35387 (local address 127.0.0.1:55612)
0621 05:23:29.793298 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.793321 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:35387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260621 05:23:29.793417 15599 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.60:35387: Network error: Client connection negotiation failed: client connection to 127.2.32.60:35387: connect: Connection refused (error 111)
I20260621 05:23:29.824576 15599 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } attempt: 1
I20260621 05:23:29.825173 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.825008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:35387 (local address 127.0.0.1:55626)
0621 05:23:29.825085 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.825139 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:35387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260621 05:23:29.825673 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.825562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:35387 (local address 127.0.0.1:55634)
0621 05:23:29.825612 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.825645 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:35387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260621 05:23:29.825837 15599 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.60:35387: Network error: Client connection negotiation failed: client connection to 127.2.32.60:35387: connect: Connection refused (error 111)
W20260621 05:23:29.834184 15681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:29.834383 15681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:29.834434 15681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:29.835594 15681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:23:29.835639 15681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:29.835654 15681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:29.835667 15681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:23:29.835680 15681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:23:29.837080 15681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:38395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:34291,127.2.32.61:37545,127.2.32.60:35387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.60.keytab
--principal=kudu/127.2.32.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.60:35387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.32.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:29.837270 15681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:29.837426 15681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:29.842043 15689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:29.842113 15687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:29.842201 15686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:29.842473 15681 server_base.cc:1034] running on GCE node
I20260621 05:23:29.842787 15681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:29.843101 15681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:29.844336 15681 hybrid_clock.cc:648] HybridClock initialized: now 1782019409844320 us; error 36 us; skew 500 ppm
Jun 21 05:23:29 dist-test-slave-b357 krb5kdc[15518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019409, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:29.847008 15681 init.cc:377] Logged in from keytab as kudu/127.2.32.60@KRBTEST.COM (short username kudu)
I20260621 05:23:29.847522 15681 webserver.cc:460] Webserver started at http://127.2.32.60:37421/ using document root <none> and password file <none>
I20260621 05:23:29.847759 15681 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:29.847827 15681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:29.847983 15681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:23:29.848971 15681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-2/data/instance:
uuid: "fb519d22ddac455d9d800a71dc2ecae5"
format_stamp: "Formatted at 2026-06-21 05:23:29 on dist-test-slave-b357"
server_key: "e12d74d3297d9fa966df402021a266df"
server_key_iv: "2ae23364837650720846841e58f76490"
server_key_version: "encryptionkey@0"
I20260621 05:23:29.849314 15681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-2/wal/instance:
uuid: "fb519d22ddac455d9d800a71dc2ecae5"
format_stamp: "Formatted at 2026-06-21 05:23:29 on dist-test-slave-b357"
server_key: "e12d74d3297d9fa966df402021a266df"
server_key_iv: "2ae23364837650720846841e58f76490"
server_key_version: "encryptionkey@0"
I20260621 05:23:29.850607 15681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:23:29.851810 15696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:29.852052 15681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:23:29.852166 15681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-2/wal
uuid: "fb519d22ddac455d9d800a71dc2ecae5"
format_stamp: "Formatted at 2026-06-21 05:23:29 on dist-test-slave-b357"
server_key: "e12d74d3297d9fa966df402021a266df"
server_key_iv: "2ae23364837650720846841e58f76490"
server_key_version: "encryptionkey@0"
I20260621 05:23:29.852246 15681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:29.860812 15674 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } attempt: 2
I20260621 05:23:29.861393 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.861228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:35387 (local address 127.0.0.1:55650)
0621 05:23:29.861303 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.861350 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:35387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260621 05:23:29.861879 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.861756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:35387 (local address 127.0.0.1:55660)
0621 05:23:29.861817 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.861853 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:35387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260621 05:23:29.862149 15674 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.60:35387: Network error: Client connection negotiation failed: client connection to 127.2.32.60:35387: connect: Connection refused (error 111)
I20260621 05:23:29.862475 15681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:29.864993 15681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:29.865191 15681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:29.869534 15681 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.60:35387
I20260621 05:23:29.869565 15748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.60:35387 every 8 connection(s)
I20260621 05:23:29.869937 15681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-2/data/info.pb
I20260621 05:23:29.870816 15749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:23:29.872382 15749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:23:29.875335 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 15681
I20260621 05:23:29.875483 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782019225111293-2176-0/minicluster-data/master-2/wal/instance
I20260621 05:23:29.875669 2176 external_mini_cluster.cc:1428] Setting key cb075ef90357b5834cf56a0a0b884cf5
Jun 21 05:23:29 dist-test-slave-b357 krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019409, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.60@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
Jun 21 05:23:29 dist-test-slave-b357 krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.60@KRBTEST.COM
I20260621 05:23:29.883860 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.873112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.2.32.62:34291)
0621 05:23:29.873204 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:29.873206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:29.873381 (+ 175us) server_negotiation.cc:407] Connection header received
0621 05:23:29.873875 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:29.873876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:29.873889 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:29.873940 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:29.874550 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.875038 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.876458 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.876791 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.876878 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.880239 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:29.880242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:29.880243 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:29.880253 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:29.882015 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.882551 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.882553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.882553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.882622 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.883026 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.883028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.883028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.883228 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:29.883314 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:29.883691 (+ 377us) server_negotiation.cc:299] Negotiation successful
0621 05:23:29.883790 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 05:23:29.884207 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.873069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:34291 (local address 127.0.0.1:32920)
0621 05:23:29.873254 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.873287 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:29.873752 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:29.874008 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:29.874011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:29.874253 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:29.874448 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:29.874452 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.875167 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.875169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:29.876328 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:29.876331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.876992 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.876994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:29.877273 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.878196 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:29.878207 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:29.880118 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:29.882167 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:29.882186 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:29.882188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:29.882450 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:29.882743 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:29.882744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:29.882745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:29.882914 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:29.883329 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:29.883335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:29.883520 (+ 185us) client_negotiation.cc:769] Sending connection context
0621 05:23:29.883663 (+ 143us) client_negotiation.cc:240] Negotiation successful
0621 05:23:29.883916 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
I20260621 05:23:29.885058 15749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:23:29.886747 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.876105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.2.32.60:35387)
0621 05:23:29.876295 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:29.876299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:29.876318 (+ 19us) server_negotiation.cc:407] Connection header received
0621 05:23:29.876339 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:29.876342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:29.876362 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:29.876435 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:29.877156 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.878538 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.879663 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.880137 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.880282 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.883163 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:29.883170 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:29.883170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:29.883178 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:29.885345 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.885853 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.885856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.885856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.885925 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.886130 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.886131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.886131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.886291 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:29.886480 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:29.886525 (+ 45us) server_negotiation.cc:299] Negotiation successful
0621 05:23:29.886615 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":117,"threads_started":1}
Jun 21 05:23:29 dist-test-slave-b357 krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019409, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.60@KRBTEST.COM for kudu/127.2.32.61@KRBTEST.COM
I20260621 05:23:29.892333 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.885414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42264 (local address 127.2.32.61:37545)
0621 05:23:29.885504 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:29.885506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:29.885514 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:23:29.885542 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:29.885542 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:29.885550 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:29.885612 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:29.885886 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.886333 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.887091 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.887333 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.887430 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.889282 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:29.889284 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:29.889284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:29.889291 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:29.891077 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.891492 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.891493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.891494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.891556 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.891781 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.891782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.891782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.891907 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:29.891984 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:29.892145 (+ 161us) server_negotiation.cc:299] Negotiation successful
0621 05:23:29.892229 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 05:23:29.892333 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.885324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:37545 (local address 127.0.0.1:42264)
0621 05:23:29.885388 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.885399 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:29.885441 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:29.885617 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:29.885617 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:29.885773 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:29.885808 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:29.885810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.886412 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.886413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:29.887014 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:29.887016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.887486 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.887486 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:29.887578 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.887998 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:29.888003 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:29.889196 (+ 1193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:29.891200 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:29.891201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:29.891202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:29.891392 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:29.891629 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:29.891630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:29.891630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:29.891714 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:29.891982 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:29.891983 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:29.892031 (+ 48us) client_negotiation.cc:769] Sending connection context
0621 05:23:29.892117 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 05:23:29.892229 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260621 05:23:29.892868 15749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:23:29 dist-test-slave-b357 krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:29.896406 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.888012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32936 (local address 127.2.32.62:34291)
0621 05:23:29.888080 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:29.888083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:29.888092 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:23:29.888113 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:29.888114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:29.888123 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:29.888190 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:29.888743 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.889345 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.890264 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.890722 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.890816 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.893552 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:29.893554 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:29.893554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:29.893562 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:29.895221 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.895586 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.895587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.895588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.895656 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.895866 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.895866 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.895867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.896011 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:29.896180 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:29.896283 (+ 103us) server_negotiation.cc:299] Negotiation successful
0621 05:23:29.896343 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Jun 21 05:23:29 dist-test-slave-b357 krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019409, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.60@KRBTEST.COM for kudu/127.2.32.60@KRBTEST.COM
I20260621 05:23:29.900843 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.893131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:35387 (local address 127.0.0.1:55682)
0621 05:23:29.893192 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.893203 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:29.893240 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:29.893413 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:29.893414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:29.893553 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:29.893596 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:29.893597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.894653 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.894654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:29.895227 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:29.895228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.895634 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.895635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:29.895676 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.896105 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:29.896110 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:29.897566 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:29.899614 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:29.899614 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:29.899615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:29.899783 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:29.900047 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:29.900048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:29.900048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:29.900113 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:29.900523 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:29.900525 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:29.900566 (+ 41us) client_negotiation.cc:769] Sending connection context
0621 05:23:29.900637 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 05:23:29.900771 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260621 05:23:29.900981 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.893205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55682 (local address 127.2.32.60:35387)
0621 05:23:29.893248 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:29.893250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:29.893261 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:23:29.893292 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:29.893292 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:29.893306 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:29.893377 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:29.893695 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.894536 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.895301 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.895540 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.895624 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.897675 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:29.897676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:29.897676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:29.897682 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:29.899501 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.899898 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.899898 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.899901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.899958 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.900198 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.900200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.900200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.900382 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:29.900482 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:29.900737 (+ 255us) server_negotiation.cc:299] Negotiation successful
0621 05:23:29.900845 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:23:29.901435 15599 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } attempt: 2
Jun 21 05:23:29 dist-test-slave-b357 krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.61@KRBTEST.COM
I20260621 05:23:29.903228 15749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5: Bootstrap starting.
I20260621 05:23:29.904381 15749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5: Neither blocks nor log segments found. Creating new log.
I20260621 05:23:29.904690 15749 log.cc:824] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:29.905527 15749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5: No bootstrap required, opened a new log
Jun 21 05:23:29 dist-test-slave-b357 krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019409, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for kudu/127.2.32.60@KRBTEST.COM
I20260621 05:23:29.906260 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.897457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42280 (local address 127.2.32.61:37545)
0621 05:23:29.897511 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:29.897513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:29.897523 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:23:29.897591 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:29.897591 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:29.897602 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:29.897658 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:29.898289 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.898763 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.899530 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.899759 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.899849 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.902705 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:29.902707 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:29.902707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:29.902716 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:29.904723 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.905145 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.905146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.905147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.905215 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.905468 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.905469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.905469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.905617 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:29.905814 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:29.906083 (+ 269us) server_negotiation.cc:299] Negotiation successful
0621 05:23:29.906171 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 05:23:29.908084 15749 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } } peers { permanent_uuid: "d51128e98e7645309c8009da172f6c39" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } } peers { permanent_uuid: "fb519d22ddac455d9d800a71dc2ecae5" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } }
I20260621 05:23:29.908254 15749 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:23:29.908283 15749 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb519d22ddac455d9d800a71dc2ecae5, State: Initialized, Role: FOLLOWER
I20260621 05:23:29.908401 15749 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } } peers { permanent_uuid: "d51128e98e7645309c8009da172f6c39" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } } peers { permanent_uuid: "fb519d22ddac455d9d800a71dc2ecae5" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } }
I20260621 05:23:29.908552 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:23:29.908696 15749 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:23:29.909312 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.901815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55690 (local address 127.2.32.60:35387)
0621 05:23:29.902009 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:29.902011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:29.902018 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:23:29.902037 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:29.902037 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:29.902047 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:29.902429 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:29.902461 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.903014 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.903898 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.904152 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.904238 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.906542 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:29.906543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:29.906543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:29.906551 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:29.908096 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.908507 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.908508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.908509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.908568 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.908771 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.908771 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.908772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.908903 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:29.908980 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:29.909140 (+ 160us) server_negotiation.cc:299] Negotiation successful
0621 05:23:29.909228 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260621 05:23:29.909317 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.901765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:35387 (local address 127.0.0.1:55690)
0621 05:23:29.901824 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.901835 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:29.901879 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:29.902128 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:29.902129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:29.902302 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:29.902354 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:29.902356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.903171 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.903172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:29.903795 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:29.903798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.904234 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.904235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:29.904296 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.904714 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:29.904718 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:29.906439 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:29.908184 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:29.908185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:29.908186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:29.908443 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:29.908639 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:29.908640 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:29.908640 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:29.908699 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:29.908985 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:29.908986 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:29.909033 (+ 47us) client_negotiation.cc:769] Sending connection context
0621 05:23:29.909117 (+ 84us) client_negotiation.cc:240] Negotiation successful
0621 05:23:29.909223 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260621 05:23:29.911702 15599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266: Bootstrap starting.
I20260621 05:23:29.912429 15599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266: Neither blocks nor log segments found. Creating new log.
I20260621 05:23:29.912726 15599 log.cc:824] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:29.910400 15755 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } } peers { permanent_uuid: "d51128e98e7645309c8009da172f6c39" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } } peers { permanent_uuid: "fb519d22ddac455d9d800a71dc2ecae5" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } } }
W20260621 05:23:29.910708 15771 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:23:29.913123 15771 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:23:29.913118 15755 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:23:29.915810 15599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266: No bootstrap required, opened a new log
I20260621 05:23:29.913563 15681 master_runner.cc:186] Error getting master registration for 127.2.32.62:34291: OK, instance_id { permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" instance_seqno: 1782019409554623 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 05:23:29.917865 15681 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 05:23:29.917796 15599 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } } peers { permanent_uuid: "d51128e98e7645309c8009da172f6c39" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } } peers { permanent_uuid: "fb519d22ddac455d9d800a71dc2ecae5" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } }
I20260621 05:23:29.917920 15599 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:23:29.917961 15599 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff960a5759954f1aa7dac71c333dd266, State: Initialized, Role: FOLLOWER
I20260621 05:23:29.918105 15599 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } } peers { permanent_uuid: "d51128e98e7645309c8009da172f6c39" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } } peers { permanent_uuid: "fb519d22ddac455d9d800a71dc2ecae5" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } }
I20260621 05:23:29.918416 15599 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:23:29.919050 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.910054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.2.32.62:34291)
0621 05:23:29.910105 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:29.910107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:29.910222 (+ 115us) server_negotiation.cc:407] Connection header received
0621 05:23:29.910326 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:29.910327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:29.910337 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:29.910368 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:29.911173 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.911863 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.913077 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.913351 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.913391 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.915043 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:29.915045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:29.915045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:29.915052 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:29.916943 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.918277 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.918278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.918279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.918341 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.918536 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.918537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.918537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.918653 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:29.918759 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:29.918870 (+ 111us) server_negotiation.cc:299] Negotiation successful
0621 05:23:29.918941 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260621 05:23:29.920462 15787 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:23:29.920517 15787 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:23:29.918376 15776 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } } peers { permanent_uuid: "d51128e98e7645309c8009da172f6c39" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } } peers { permanent_uuid: "fb519d22ddac455d9d800a71dc2ecae5" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } } }
I20260621 05:23:29.920961 15776 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:23:29.921237 15531 master_runner.cc:186] Error getting master registration for 127.2.32.61:37545: OK, instance_id { permanent_uuid: "d51128e98e7645309c8009da172f6c39" instance_seqno: 1782019409714095 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 05:23:29.923862 15531 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 05:23:29.924746 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.910659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42296 (local address 127.2.32.61:37545)
0621 05:23:29.910694 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:29.910696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:29.910985 (+ 289us) server_negotiation.cc:407] Connection header received
0621 05:23:29.911079 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:29.911080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:29.911089 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:29.911116 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:29.911857 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.912523 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.913571 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.913869 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.913917 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.915377 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:29.915378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:29.915379 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:29.915385 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:29.917626 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.921028 (+ 3402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.921028 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.921029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.921084 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.924088 (+ 3004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.924089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.924089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.924233 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:29.924340 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:29.924541 (+ 201us) server_negotiation.cc:299] Negotiation successful
0621 05:23:29.924618 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:23:29.926127 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.910405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55692 (local address 127.2.32.60:35387)
0621 05:23:29.910475 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:29.910477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:29.919082 (+ 8605us) server_negotiation.cc:407] Connection header received
0621 05:23:29.919230 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:29.919231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:29.919243 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:29.919275 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:29.919750 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.920428 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.921394 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.921670 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.921729 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.923023 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:29.923025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:29.923025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:29.923031 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:29.924808 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.925371 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.925373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.925373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.925434 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.925643 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.925644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.925645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.925756 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:29.925867 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:29.925973 (+ 106us) server_negotiation.cc:299] Negotiation successful
0621 05:23:29.926040 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 05:23:29.953332 15674 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } attempt: 3
Jun 21 05:23:29 dist-test-slave-b357 krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019409, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.61@KRBTEST.COM for kudu/127.2.32.60@KRBTEST.COM
I20260621 05:23:29.961614 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.953615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:35387 (local address 127.0.0.1:55696)
0621 05:23:29.953665 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:29.953676 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:29.953719 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:29.953869 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:29.953870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:29.954052 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:29.954107 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:29.954109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.955038 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.955039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:29.955946 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:29.955948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.956393 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.956394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:29.956454 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.956972 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:29.956977 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:29.958755 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:29.960536 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:29.960537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:29.960537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:29.960764 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:29.960969 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:29.960970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:29.960970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:29.961034 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:29.961287 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:29.961288 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:29.961351 (+ 63us) client_negotiation.cc:769] Sending connection context
0621 05:23:29.961421 (+ 70us) client_negotiation.cc:240] Negotiation successful
0621 05:23:29.961514 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260621 05:23:29.961889 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:29.953697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.2.32.60:35387)
0621 05:23:29.953746 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:29.953749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:29.953756 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:23:29.953777 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:29.953778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:29.953790 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:29.953851 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:29.954186 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.954959 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.956021 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:29.956311 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:29.956395 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:29.958837 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:29.958839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:29.958839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:29.958846 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:29.960448 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.960840 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.960841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.960841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.960897 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:29.961098 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:29.961099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:29.961099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:29.961219 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:29.961284 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:29.961735 (+ 451us) server_negotiation.cc:299] Negotiation successful
0621 05:23:29.961804 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260621 05:23:29.963708 15674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39: Bootstrap starting.
I20260621 05:23:29.964412 15674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39: Neither blocks nor log segments found. Creating new log.
I20260621 05:23:29.964745 15674 log.cc:824] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:29.965600 15674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39: No bootstrap required, opened a new log
I20260621 05:23:29.967648 15674 raft_consensus.cc:348] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } } peers { permanent_uuid: "d51128e98e7645309c8009da172f6c39" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } } peers { permanent_uuid: "fb519d22ddac455d9d800a71dc2ecae5" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } }
I20260621 05:23:29.967794 15674 raft_consensus.cc:374] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:23:29.967836 15674 raft_consensus.cc:729] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d51128e98e7645309c8009da172f6c39, State: Initialized, Role: FOLLOWER
I20260621 05:23:29.967973 15674 consensus_queue.cc:260] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } } peers { permanent_uuid: "d51128e98e7645309c8009da172f6c39" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } } peers { permanent_uuid: "fb519d22ddac455d9d800a71dc2ecae5" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } }
I20260621 05:23:29.968283 15674 sys_catalog.cc:564] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:23:29.968246 15790 sys_catalog.cc:455] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } } peers { permanent_uuid: "d51128e98e7645309c8009da172f6c39" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } } peers { permanent_uuid: "fb519d22ddac455d9d800a71dc2ecae5" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } } }
I20260621 05:23:29.969981 15790 sys_catalog.cc:458] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [sys.catalog]: This master's current role is: FOLLOWER
W20260621 05:23:29.970391 15801 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:23:29.970443 15801 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:23:29.972107 15606 master_runner.cc:263] No leader master found from master d51128e98e7645309c8009da172f6c39
I20260621 05:23:29.972155 15606 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 05:23:29.983309 15790 raft_consensus.cc:482] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:23:29.983492 15790 raft_consensus.cc:504] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } } peers { permanent_uuid: "d51128e98e7645309c8009da172f6c39" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } } peers { permanent_uuid: "fb519d22ddac455d9d800a71dc2ecae5" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } }
I20260621 05:23:29.984004 15573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d51128e98e7645309c8009da172f6c39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff960a5759954f1aa7dac71c333dd266" is_pre_election: true
I20260621 05:23:29.984136 15573 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d51128e98e7645309c8009da172f6c39 in term 0.
I20260621 05:23:29.984632 15624 leader_election.cc:302] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d51128e98e7645309c8009da172f6c39, ff960a5759954f1aa7dac71c333dd266; no voters:
I20260621 05:23:29.985137 15723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d51128e98e7645309c8009da172f6c39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb519d22ddac455d9d800a71dc2ecae5" is_pre_election: true
I20260621 05:23:29.985263 15723 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d51128e98e7645309c8009da172f6c39 in term 0.
I20260621 05:23:29.985517 15790 leader_election.cc:288] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff960a5759954f1aa7dac71c333dd266 (127.2.32.62:34291), fb519d22ddac455d9d800a71dc2ecae5 (127.2.32.60:35387)
I20260621 05:23:29.985631 15790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:23:29.985723 15790 raft_consensus.cc:482] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:23:29.985792 15790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:23:29.986720 15790 raft_consensus.cc:504] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } } peers { permanent_uuid: "d51128e98e7645309c8009da172f6c39" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } } peers { permanent_uuid: "fb519d22ddac455d9d800a71dc2ecae5" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } }
I20260621 05:23:29.987002 15573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d51128e98e7645309c8009da172f6c39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff960a5759954f1aa7dac71c333dd266"
I20260621 05:23:29.987075 15573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:23:29.987993 15573 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d51128e98e7645309c8009da172f6c39 in term 1.
I20260621 05:23:29.988206 15624 leader_election.cc:302] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d51128e98e7645309c8009da172f6c39, ff960a5759954f1aa7dac71c333dd266; no voters:
I20260621 05:23:29.988461 15723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d51128e98e7645309c8009da172f6c39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb519d22ddac455d9d800a71dc2ecae5"
I20260621 05:23:29.988530 15723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:23:29.989301 15723 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d51128e98e7645309c8009da172f6c39 in term 1.
I20260621 05:23:29.990361 15790 leader_election.cc:288] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [CANDIDATE]: Term 1 election: Requested vote from peers ff960a5759954f1aa7dac71c333dd266 (127.2.32.62:34291), fb519d22ddac455d9d800a71dc2ecae5 (127.2.32.60:35387)
I20260621 05:23:29.990470 15790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:23:29.990547 15790 raft_consensus.cc:686] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [term 1 LEADER]: Becoming Leader. State: Replica: d51128e98e7645309c8009da172f6c39, State: Running, Role: LEADER
I20260621 05:23:29.990661 15790 consensus_queue.cc:237] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } } peers { permanent_uuid: "d51128e98e7645309c8009da172f6c39" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } } peers { permanent_uuid: "fb519d22ddac455d9d800a71dc2ecae5" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } }
I20260621 05:23:29.991160 15790 sys_catalog.cc:455] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d51128e98e7645309c8009da172f6c39. Latest consensus state: current_term: 1 leader_uuid: "d51128e98e7645309c8009da172f6c39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } } peers { permanent_uuid: "d51128e98e7645309c8009da172f6c39" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } } peers { permanent_uuid: "fb519d22ddac455d9d800a71dc2ecae5" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } } }
I20260621 05:23:29.991287 15790 sys_catalog.cc:458] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [sys.catalog]: This master's current role is: LEADER
I20260621 05:23:29.991997 15805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:23:29.992236 15805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:23:29.993750 15723 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5 [term 1 FOLLOWER]: Refusing update from remote peer d51128e98e7645309c8009da172f6c39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:23:29.994119 15803 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb519d22ddac455d9d800a71dc2ecae5" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:23:29.995334 15755 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d51128e98e7645309c8009da172f6c39. Latest consensus state: current_term: 1 leader_uuid: "d51128e98e7645309c8009da172f6c39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } } peers { permanent_uuid: "d51128e98e7645309c8009da172f6c39" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } } peers { permanent_uuid: "fb519d22ddac455d9d800a71dc2ecae5" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } } }
I20260621 05:23:29.995450 15755 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:23:29.995589 15573 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266 [term 1 FOLLOWER]: Refusing update from remote peer d51128e98e7645309c8009da172f6c39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:23:29.995919 15803 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:23:29.997020 15776 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d51128e98e7645309c8009da172f6c39. Latest consensus state: current_term: 1 leader_uuid: "d51128e98e7645309c8009da172f6c39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } } peers { permanent_uuid: "d51128e98e7645309c8009da172f6c39" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } } peers { permanent_uuid: "fb519d22ddac455d9d800a71dc2ecae5" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } } }
I20260621 05:23:29.997098 15776 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:23:29.998023 15803 sys_catalog.cc:455] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d51128e98e7645309c8009da172f6c39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } } peers { permanent_uuid: "d51128e98e7645309c8009da172f6c39" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } } peers { permanent_uuid: "fb519d22ddac455d9d800a71dc2ecae5" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } } }
I20260621 05:23:29.998108 15803 sys_catalog.cc:458] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [sys.catalog]: This master's current role is: LEADER
I20260621 05:23:29.998512 15776 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d51128e98e7645309c8009da172f6c39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } } peers { permanent_uuid: "d51128e98e7645309c8009da172f6c39" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } } peers { permanent_uuid: "fb519d22ddac455d9d800a71dc2ecae5" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } } }
I20260621 05:23:29.998585 15776 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:23:29.998832 15806 mvcc.cc:205] Tried to move back new op lower bound from 7299151503332225024 to 7299151503322738688. Current Snapshot: MvccSnapshot[applied={T|T < 7299151503332225024}]
I20260621 05:23:29.999394 15809 mvcc.cc:205] Tried to move back new op lower bound from 7299151503332225024 to 7299151503322738688. Current Snapshot: MvccSnapshot[applied={T|T < 7299151503332225024}]
I20260621 05:23:30.000226 15790 sys_catalog.cc:455] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d51128e98e7645309c8009da172f6c39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } } peers { permanent_uuid: "d51128e98e7645309c8009da172f6c39" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } } peers { permanent_uuid: "fb519d22ddac455d9d800a71dc2ecae5" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } } }
I20260621 05:23:30.000298 15790 sys_catalog.cc:458] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39 [sys.catalog]: This master's current role is: LEADER
I20260621 05:23:30.000646 15755 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d51128e98e7645309c8009da172f6c39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff960a5759954f1aa7dac71c333dd266" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34291 } } peers { permanent_uuid: "d51128e98e7645309c8009da172f6c39" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 37545 } } peers { permanent_uuid: "fb519d22ddac455d9d800a71dc2ecae5" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 35387 } } }
I20260621 05:23:30.000730 15755 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:23:30.001348 15805 catalog_manager.cc:1347] Generated new cluster ID: aac2200ea56d409aa5f917a06d7d4d51
I20260621 05:23:30.001392 15805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:23:30.014437 15807 mvcc.cc:205] Tried to move back new op lower bound from 7299151503332225024 to 7299151503322738688. Current Snapshot: MvccSnapshot[applied={T|T < 7299151503332225024}]
I20260621 05:23:30.017457 15805 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:23:30.018280 15805 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:23:30.030407 15805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d51128e98e7645309c8009da172f6c39: Generated new TSK 0
I20260621 05:23:30.030646 15805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:23:30.913620 15771 catalog_manager.cc:1259] Loaded cluster ID: aac2200ea56d409aa5f917a06d7d4d51
I20260621 05:23:30.913708 15771 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5: loading cluster ID for follower catalog manager: success
I20260621 05:23:30.914561 15771 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5: acquiring CA information for follower catalog manager: success
I20260621 05:23:30.914880 15771 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fb519d22ddac455d9d800a71dc2ecae5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:23:30.920873 15787 catalog_manager.cc:1259] Loaded cluster ID: aac2200ea56d409aa5f917a06d7d4d51
I20260621 05:23:30.920928 15787 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266: loading cluster ID for follower catalog manager: success
I20260621 05:23:30.921720 15787 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266: acquiring CA information for follower catalog manager: success
I20260621 05:23:30.922003 15787 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ff960a5759954f1aa7dac71c333dd266: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:23:31.086768 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 15531
I20260621 05:23:31.092365 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 15606
I20260621 05:23:31.097283 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 15681
2026-06-21T05:23:31Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2872 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:23:31 dist-test-slave-b357 krb5kdc[15822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:23:31 dist-test-slave-b357 krb5kdc[15822](info): set up 2 sockets
Jun 21 05:23:31 dist-test-slave-b357 krb5kdc[15822](info): commencing operation
krb5kdc: starting...
W20260621 05:23:32.151340 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:23:32 dist-test-slave-b357 krb5kdc[15822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:23:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:23:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:23:32.283840 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:41529
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:46765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:41529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:23:32.375043 15835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:32.375379 15835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:32.375422 15835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:32.376538 15835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:23:32.376607 15835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:32.376634 15835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:32.376660 15835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:23:32.376693 15835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:23:32.378212 15835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:46765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:41529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:41529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:32.378525 15835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:32.378756 15835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:32.384524 15843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:32.384509 15841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:32.384510 15840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:32.384938 15835 server_base.cc:1034] running on GCE node
I20260621 05:23:32.385212 15835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:32.385581 15835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:32.386791 15835 hybrid_clock.cc:648] HybridClock initialized: now 1782019412386770 us; error 43 us; skew 500 ppm
Jun 21 05:23:32 dist-test-slave-b357 krb5kdc[15822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019412, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:32.389706 15835 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:23:32.390223 15835 webserver.cc:460] Webserver started at http://127.2.32.62:41369/ using document root <none> and password file <none>
I20260621 05:23:32.390444 15835 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:32.390512 15835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:32.390805 15835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:23:32.391966 15835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "70af84ec04214fbe963ee6cec8b30827"
format_stamp: "Formatted at 2026-06-21 05:23:32 on dist-test-slave-b357"
server_key: "89a266eeb48d76bafb68c8bbcb2d34da"
server_key_iv: "9d4066e6761ed52869ceb8681b020192"
server_key_version: "encryptionkey@0"
I20260621 05:23:32.392333 15835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "70af84ec04214fbe963ee6cec8b30827"
format_stamp: "Formatted at 2026-06-21 05:23:32 on dist-test-slave-b357"
server_key: "89a266eeb48d76bafb68c8bbcb2d34da"
server_key_iv: "9d4066e6761ed52869ceb8681b020192"
server_key_version: "encryptionkey@0"
I20260621 05:23:32.393833 15835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:23:32.394907 15850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:32.395313 15835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:32.395515 15835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "70af84ec04214fbe963ee6cec8b30827"
format_stamp: "Formatted at 2026-06-21 05:23:32 on dist-test-slave-b357"
server_key: "89a266eeb48d76bafb68c8bbcb2d34da"
server_key_iv: "9d4066e6761ed52869ceb8681b020192"
server_key_version: "encryptionkey@0"
I20260621 05:23:32.395632 15835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:32.415604 15835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:32.418918 15835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:32.419101 15835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:32.424235 15835 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:41529
I20260621 05:23:32.424279 15902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:41529 every 8 connection(s)
I20260621 05:23:32.424825 15835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:23:32.425580 15903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:23:32.428062 15903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827: Bootstrap starting.
I20260621 05:23:32.428680 15903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827: Neither blocks nor log segments found. Creating new log.
I20260621 05:23:32.428944 15903 log.cc:824] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:32.429625 15903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827: No bootstrap required, opened a new log
I20260621 05:23:32.430842 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 15835
I20260621 05:23:32.430962 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:23:32.431187 2176 external_mini_cluster.cc:1428] Setting key a3884cc49ea75c90d142e291e1071ef0
I20260621 05:23:32.431779 15903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70af84ec04214fbe963ee6cec8b30827" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 41529 } }
I20260621 05:23:32.431958 15903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:23:32.431985 15903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70af84ec04214fbe963ee6cec8b30827, State: Initialized, Role: FOLLOWER
I20260621 05:23:32.432130 15903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70af84ec04214fbe963ee6cec8b30827" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 41529 } }
I20260621 05:23:32.432206 15903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:23:32.432228 15903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:23:32.432294 15903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:23:32.432835 15903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70af84ec04214fbe963ee6cec8b30827" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 41529 } }
I20260621 05:23:32.432916 15903 leader_election.cc:302] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70af84ec04214fbe963ee6cec8b30827; no voters:
I20260621 05:23:32.433091 15903 leader_election.cc:288] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:23:32.433135 15908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:23:32.433306 15908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827 [term 1 LEADER]: Becoming Leader. State: Replica: 70af84ec04214fbe963ee6cec8b30827, State: Running, Role: LEADER
I20260621 05:23:32.433382 15903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:23:32.433447 15908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70af84ec04214fbe963ee6cec8b30827" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 41529 } }
I20260621 05:23:32.433808 15908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70af84ec04214fbe963ee6cec8b30827. Latest consensus state: current_term: 1 leader_uuid: "70af84ec04214fbe963ee6cec8b30827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70af84ec04214fbe963ee6cec8b30827" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 41529 } } }
I20260621 05:23:32.433817 15910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70af84ec04214fbe963ee6cec8b30827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70af84ec04214fbe963ee6cec8b30827" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 41529 } } }
I20260621 05:23:32.433930 15908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827 [sys.catalog]: This master's current role is: LEADER
I20260621 05:23:32.433941 15910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827 [sys.catalog]: This master's current role is: LEADER
I20260621 05:23:32.435047 15914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:23:32.435534 15914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:23:32.436861 15914 catalog_manager.cc:1347] Generated new cluster ID: 81d70b96634545888a8c56d5852a1048
I20260621 05:23:32.436916 15914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:23:32 dist-test-slave-b357 krb5kdc[15822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:32.444002 15914 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:23:32.445034 15914 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:23:32.446070 15907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:32.431710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42522 (local address 127.2.32.62:41529)
0621 05:23:32.431883 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:32.431891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:32.431916 (+ 25us) server_negotiation.cc:407] Connection header received
0621 05:23:32.432064 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:32.432071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:32.432131 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:32.432293 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:32.433504 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:32.434853 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:32.436091 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:32.436890 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:32.437053 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:32.440723 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:32.440730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:32.440731 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:32.440749 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:32.443887 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:32.444458 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:32.444462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:32.444463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:32.444588 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:32.444821 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:32.444822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:32.444822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:32.445176 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:32.445331 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:32.445632 (+ 301us) server_negotiation.cc:299] Negotiation successful
0621 05:23:32.445820 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260621 05:23:32.447513 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:23:32.450160 15914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70af84ec04214fbe963ee6cec8b30827: Generated new TSK 0
I20260621 05:23:32.450326 15914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:23:32.503916 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 15835
2026-06-21T05:23:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1405 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:23:32 dist-test-slave-b357 krb5kdc[15930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:23:32 dist-test-slave-b357 krb5kdc[15930](info): set up 2 sockets
Jun 21 05:23:32 dist-test-slave-b357 krb5kdc[15930](info): commencing operation
krb5kdc: starting...
W20260621 05:23:33.647746 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:23:33 dist-test-slave-b357 krb5kdc[15930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:23:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:23:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:23:33.795729 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:32835
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:32835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:23:33.888881 15943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:33.889124 15943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:33.889161 15943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:33.890272 15943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:23:33.890340 15943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:33.890367 15943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:33.890399 15943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:23:33.890431 15943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:23:33.891912 15943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:32835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:32835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:33.892184 15943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:33.892412 15943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:23:33.897702 15943 server_base.cc:1034] running on GCE node
W20260621 05:23:33.897522 15949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:33.897528 15951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:33.897653 15948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:33.898294 15943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:33.898638 15943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:33.899828 15943 hybrid_clock.cc:648] HybridClock initialized: now 1782019413899816 us; error 32 us; skew 500 ppm
Jun 21 05:23:33 dist-test-slave-b357 krb5kdc[15930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019413, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:33.909322 15943 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:23:33.909965 15943 webserver.cc:460] Webserver started at http://127.2.32.62:40465/ using document root <none> and password file <none>
I20260621 05:23:33.910293 15943 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:33.910382 15943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:33.910549 15943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:23:33.912418 15943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "e01df39404d24d239c5ac2c66ee4d3e0"
format_stamp: "Formatted at 2026-06-21 05:23:33 on dist-test-slave-b357"
server_key: "0b05095e79aea56f9ed3cd7bd6ad18a9"
server_key_iv: "390f04ce071e7384ced966b02868d6de"
server_key_version: "encryptionkey@0"
I20260621 05:23:33.912920 15943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "e01df39404d24d239c5ac2c66ee4d3e0"
format_stamp: "Formatted at 2026-06-21 05:23:33 on dist-test-slave-b357"
server_key: "0b05095e79aea56f9ed3cd7bd6ad18a9"
server_key_iv: "390f04ce071e7384ced966b02868d6de"
server_key_version: "encryptionkey@0"
I20260621 05:23:33.914669 15943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:23:33.915791 15958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:33.916144 15943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:23:33.916307 15943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "e01df39404d24d239c5ac2c66ee4d3e0"
format_stamp: "Formatted at 2026-06-21 05:23:33 on dist-test-slave-b357"
server_key: "0b05095e79aea56f9ed3cd7bd6ad18a9"
server_key_iv: "390f04ce071e7384ced966b02868d6de"
server_key_version: "encryptionkey@0"
I20260621 05:23:33.916426 15943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:33.926569 15943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:33.929454 15943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:33.929634 15943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:33.934566 15943 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:32835
I20260621 05:23:33.934654 16010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:32835 every 8 connection(s)
I20260621 05:23:33.935081 15943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:23:33.936442 16011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:23:33.939088 16011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0: Bootstrap starting.
I20260621 05:23:33.939816 16011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0: Neither blocks nor log segments found. Creating new log.
I20260621 05:23:33.940096 16011 log.cc:824] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:33.941062 16011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0: No bootstrap required, opened a new log
I20260621 05:23:33.942584 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 15943
I20260621 05:23:33.942687 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:23:33.942925 2176 external_mini_cluster.cc:1428] Setting key 212f237453848f45b4f9e751fc873283
I20260621 05:23:33.943447 16011 raft_consensus.cc:348] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 32835 } }
I20260621 05:23:33.943696 16011 raft_consensus.cc:374] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:23:33.943773 16011 raft_consensus.cc:729] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01df39404d24d239c5ac2c66ee4d3e0, State: Initialized, Role: FOLLOWER
I20260621 05:23:33.943931 16011 consensus_queue.cc:260] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 32835 } }
I20260621 05:23:33.944036 16011 raft_consensus.cc:388] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:23:33.944082 16011 raft_consensus.cc:482] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:23:33.944135 16011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:23:33.945508 16011 raft_consensus.cc:504] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 32835 } }
I20260621 05:23:33.945650 16011 leader_election.cc:302] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e01df39404d24d239c5ac2c66ee4d3e0; no voters:
I20260621 05:23:33.945858 16011 leader_election.cc:288] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:23:33.945917 16016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:23:33.946115 16016 raft_consensus.cc:686] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 1 LEADER]: Becoming Leader. State: Replica: e01df39404d24d239c5ac2c66ee4d3e0, State: Running, Role: LEADER
I20260621 05:23:33.946275 16011 sys_catalog.cc:564] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:23:33.946285 16016 consensus_queue.cc:237] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 32835 } }
I20260621 05:23:33.946695 16017 sys_catalog.cc:455] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e01df39404d24d239c5ac2c66ee4d3e0. Latest consensus state: current_term: 1 leader_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 32835 } } }
I20260621 05:23:33.946810 16017 sys_catalog.cc:458] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [sys.catalog]: This master's current role is: LEADER
I20260621 05:23:33.947060 16023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:23:33.947645 16023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:23:33.948099 16018 sys_catalog.cc:455] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 32835 } } }
I20260621 05:23:33.948278 16018 sys_catalog.cc:458] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [sys.catalog]: This master's current role is: LEADER
I20260621 05:23:33.949350 16023 catalog_manager.cc:1347] Generated new cluster ID: bf8cba02be7a4f5d991cb74e7384edd3
I20260621 05:23:33.949401 16023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:23:33 dist-test-slave-b357 krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:33.955845 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:33.943611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60644 (local address 127.2.32.62:32835)
0621 05:23:33.943860 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:33.943872 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:33.943904 (+ 32us) server_negotiation.cc:407] Connection header received
0621 05:23:33.944032 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:33.944038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:33.944116 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:33.944257 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:33.945186 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:33.946502 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:33.947858 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:33.948438 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:33.948550 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:33.951612 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:33.951616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:33.951617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:33.951630 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:33.953873 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:33.954242 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:33.954245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:33.954245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:33.954387 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:33.954592 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:33.954593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:33.954594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:33.954928 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:33.955120 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:33.955472 (+ 352us) server_negotiation.cc:299] Negotiation successful
0621 05:23:33.955622 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":106,"threads_started":1}
I20260621 05:23:33.968538 16023 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:23:33.969473 16023 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:23:33.975417 16023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0: Generated new TSK 0
I20260621 05:23:33.975672 16023 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
I20260621 05:23:34.013727 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:32835
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:23:34.105476 16037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:34.105703 16037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:34.105743 16037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:34.106875 16037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:34.106943 16037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:34.106988 16037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:23:34.108496 16037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:32835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:34.108757 16037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:34.108971 16037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:34.114395 16042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:34.114529 16043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:34.114593 16037 server_base.cc:1034] running on GCE node
W20260621 05:23:34.114408 16045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:34.114913 16037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:34.115214 16037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:34.116377 16037 hybrid_clock.cc:648] HybridClock initialized: now 1782019414116354 us; error 41 us; skew 500 ppm
Jun 21 05:23:34 dist-test-slave-b357 krb5kdc[15930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019414, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:34.118891 16037 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:23:34.119378 16037 webserver.cc:460] Webserver started at http://127.2.32.1:34149/ using document root <none> and password file <none>
I20260621 05:23:34.119638 16037 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:34.119707 16037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:34.119843 16037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:23:34.120937 16037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "481ca11e1b404475b9d31696996a6a33"
format_stamp: "Formatted at 2026-06-21 05:23:34 on dist-test-slave-b357"
server_key: "bdcd702c76863608cd79b2928a66363c"
server_key_iv: "c1037bba371917cf42f6f25b3246c07f"
server_key_version: "encryptionkey@0"
I20260621 05:23:34.121289 16037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "481ca11e1b404475b9d31696996a6a33"
format_stamp: "Formatted at 2026-06-21 05:23:34 on dist-test-slave-b357"
server_key: "bdcd702c76863608cd79b2928a66363c"
server_key_iv: "c1037bba371917cf42f6f25b3246c07f"
server_key_version: "encryptionkey@0"
I20260621 05:23:34.122596 16037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:23:34.123366 16052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:34.123539 16037 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:34.123620 16037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "481ca11e1b404475b9d31696996a6a33"
format_stamp: "Formatted at 2026-06-21 05:23:34 on dist-test-slave-b357"
server_key: "bdcd702c76863608cd79b2928a66363c"
server_key_iv: "c1037bba371917cf42f6f25b3246c07f"
server_key_version: "encryptionkey@0"
I20260621 05:23:34.123689 16037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:34.141228 16037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:34.144402 16037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:34.144647 16037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:34.144960 16037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:23:34.145383 16037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:23:34.145443 16037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:34.145489 16037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:23:34.145521 16037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:34.154094 16037 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:43139
I20260621 05:23:34.154135 16165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:43139 every 8 connection(s)
I20260621 05:23:34.154628 16037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:23:34.161752 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 16037
I20260621 05:23:34.161895 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:23:34.162122 2176 external_mini_cluster.cc:1428] Setting key 97e75a065cac1c22e75398b8a04c1c16
Jun 21 05:23:34 dist-test-slave-b357 krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019414, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:34.168533 16168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:34.155429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.1:45141)
0621 05:23:34.155607 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:34.155653 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:34.156421 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:34.156775 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:34.156779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:34.157612 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:34.157915 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:34.157921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.158896 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.158897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:34.160192 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:34.160195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.160762 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.160764 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:34.160947 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:34.161943 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:34.161956 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:34.164211 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:34.166682 (+ 2471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:34.166688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:34.166689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:34.166917 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:34.167257 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:34.167259 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:34.167259 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:34.167424 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:34.167823 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:34.167832 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:34.168058 (+ 226us) client_negotiation.cc:769] Sending connection context
0621 05:23:34.168191 (+ 133us) client_negotiation.cc:240] Negotiation successful
0621 05:23:34.168326 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260621 05:23:34.168583 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:34.155448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:45141 (local address 127.2.32.62:32835)
0621 05:23:34.155495 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:34.155498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:34.155761 (+ 263us) server_negotiation.cc:407] Connection header received
0621 05:23:34.156557 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:34.156558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:34.156574 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:34.156604 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:34.158065 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.158773 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.160275 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.160560 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.160636 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:34.164344 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:34.164345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:34.164346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:34.164354 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:34.166522 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:34.167015 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:34.167016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:34.167017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:34.167102 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:34.167531 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:34.167532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:34.167533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:34.167713 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:34.167794 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:34.168303 (+ 509us) server_negotiation.cc:299] Negotiation successful
0621 05:23:34.168494 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:23:34.169042 16166 heartbeater.cc:347] Connected to a master server at 127.2.32.62:32835
I20260621 05:23:34.169166 16166 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:34.169351 16166 heartbeater.cc:511] Master 127.2.32.62:32835 requested a full tablet report, sending...
I20260621 05:23:34.169906 15975 ts_manager.cc:194] Registered new tserver with Master: 481ca11e1b404475b9d31696996a6a33 (127.2.32.1:43139)
I20260621 05:23:34.170533 15975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:45141
WARNING: no policy specified for kudu/127.2.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
I20260621 05:23:34.194716 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:32835
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:23:34.276407 16171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:34.276605 16171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:34.276629 16171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:34.277997 16171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:34.278054 16171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:34.278082 16171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:23:34.279618 16171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:32835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:34.279810 16171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:34.279978 16171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:23:34.285135 16171 server_base.cc:1034] running on GCE node
W20260621 05:23:34.285137 16179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:34.285164 16177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:34.285248 16176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:34.285720 16171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:34.286042 16171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:34.287240 16171 hybrid_clock.cc:648] HybridClock initialized: now 1782019414287206 us; error 51 us; skew 500 ppm
Jun 21 05:23:34 dist-test-slave-b357 krb5kdc[15930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019414, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:34.290342 16171 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:23:34.290813 16171 webserver.cc:460] Webserver started at http://127.2.32.2:40425/ using document root <none> and password file <none>
I20260621 05:23:34.291038 16171 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:34.291106 16171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:34.291258 16171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:23:34.292312 16171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "956c8c30e25b4adf806e5cd32b30d39e"
format_stamp: "Formatted at 2026-06-21 05:23:34 on dist-test-slave-b357"
server_key: "2e6e1cbb9d0785b396ba899838bed749"
server_key_iv: "4e8d576fe664a6c282cd5934e8388af9"
server_key_version: "encryptionkey@0"
I20260621 05:23:34.292677 16171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "956c8c30e25b4adf806e5cd32b30d39e"
format_stamp: "Formatted at 2026-06-21 05:23:34 on dist-test-slave-b357"
server_key: "2e6e1cbb9d0785b396ba899838bed749"
server_key_iv: "4e8d576fe664a6c282cd5934e8388af9"
server_key_version: "encryptionkey@0"
I20260621 05:23:34.294018 16171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:23:34.294864 16186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:34.295121 16171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:34.295259 16171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "956c8c30e25b4adf806e5cd32b30d39e"
format_stamp: "Formatted at 2026-06-21 05:23:34 on dist-test-slave-b357"
server_key: "2e6e1cbb9d0785b396ba899838bed749"
server_key_iv: "4e8d576fe664a6c282cd5934e8388af9"
server_key_version: "encryptionkey@0"
I20260621 05:23:34.295354 16171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:34.312312 16171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:34.314623 16171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:34.314810 16171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:34.315068 16171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:23:34.315526 16171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:23:34.315611 16171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:34.315655 16171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:23:34.315714 16171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:34.321926 16171 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:42189
I20260621 05:23:34.321979 16299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:42189 every 8 connection(s)
I20260621 05:23:34.322304 16171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:23:34.330008 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 16171
I20260621 05:23:34.330160 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:23:34.330402 2176 external_mini_cluster.cc:1428] Setting key 04443691b72daf99bc90a3b21294fd63
Jun 21 05:23:34 dist-test-slave-b357 krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019414, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:34.334350 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:34.323540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:35299 (local address 127.2.32.62:32835)
0621 05:23:34.323587 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:34.323589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:34.324125 (+ 536us) server_negotiation.cc:407] Connection header received
0621 05:23:34.325244 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:34.325246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:34.325262 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:34.325355 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:34.326430 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.326966 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.328187 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.328387 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.328456 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:34.331499 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:34.331501 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:34.331501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:34.331510 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:34.332960 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:34.333391 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:34.333392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:34.333393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:34.333452 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:34.333704 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:34.333705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:34.333706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:34.333817 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:34.333894 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:34.334162 (+ 268us) server_negotiation.cc:299] Negotiation successful
0621 05:23:34.334254 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:23:34.334554 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:34.323601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.2:35299)
0621 05:23:34.323877 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:34.323928 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:34.325045 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:34.325463 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:34.325467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:34.325932 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:34.326285 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:34.326296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.327107 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.327108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:34.328089 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:34.328092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.328526 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.328527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:34.328708 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:34.329581 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:34.329590 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:34.331349 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:34.333058 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:34.333064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:34.333066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:34.333296 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:34.333519 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:34.333520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:34.333521 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:34.333635 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:34.333885 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:34.333891 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:34.334031 (+ 140us) client_negotiation.cc:769] Sending connection context
0621 05:23:34.334151 (+ 120us) client_negotiation.cc:240] Negotiation successful
0621 05:23:34.334314 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":134,"threads_started":1}
I20260621 05:23:34.335090 16300 heartbeater.cc:347] Connected to a master server at 127.2.32.62:32835
I20260621 05:23:34.335217 16300 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:34.335395 16300 heartbeater.cc:511] Master 127.2.32.62:32835 requested a full tablet report, sending...
I20260621 05:23:34.335793 15975 ts_manager.cc:194] Registered new tserver with Master: 956c8c30e25b4adf806e5cd32b30d39e (127.2.32.2:42189)
I20260621 05:23:34.336354 15975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:35299
WARNING: no policy specified for kudu/127.2.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
I20260621 05:23:34.362731 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:32835
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:23:34.456851 16305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:34.457083 16305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:34.457116 16305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:34.458968 16305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:34.459026 16305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:34.459055 16305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:23:34.460644 16305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:32835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:34.460839 16305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:34.461016 16305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:34.466765 16313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:34.466912 16310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:34.466761 16311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:34.468212 16305 server_base.cc:1034] running on GCE node
I20260621 05:23:34.468851 16305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:34.469129 16305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:34.470453 16305 hybrid_clock.cc:648] HybridClock initialized: now 1782019414470412 us; error 57 us; skew 500 ppm
Jun 21 05:23:34 dist-test-slave-b357 krb5kdc[15930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019414, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:34.474080 16305 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:23:34.475088 16305 webserver.cc:460] Webserver started at http://127.2.32.3:45583/ using document root <none> and password file <none>
I20260621 05:23:34.475396 16305 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:34.475512 16305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:34.475816 16305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:23:34.476941 16305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "63d927b55814423a9280aae6fdc9083e"
format_stamp: "Formatted at 2026-06-21 05:23:34 on dist-test-slave-b357"
server_key: "7a8ffabf4e855250c06dbfd6afc27e0d"
server_key_iv: "bccdb0bcf4c4defd0888e04f67b541e9"
server_key_version: "encryptionkey@0"
I20260621 05:23:34.477319 16305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "63d927b55814423a9280aae6fdc9083e"
format_stamp: "Formatted at 2026-06-21 05:23:34 on dist-test-slave-b357"
server_key: "7a8ffabf4e855250c06dbfd6afc27e0d"
server_key_iv: "bccdb0bcf4c4defd0888e04f67b541e9"
server_key_version: "encryptionkey@0"
I20260621 05:23:34.479215 16305 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260621 05:23:34.480203 16320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:34.480479 16305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:34.480611 16305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "63d927b55814423a9280aae6fdc9083e"
format_stamp: "Formatted at 2026-06-21 05:23:34 on dist-test-slave-b357"
server_key: "7a8ffabf4e855250c06dbfd6afc27e0d"
server_key_iv: "bccdb0bcf4c4defd0888e04f67b541e9"
server_key_version: "encryptionkey@0"
I20260621 05:23:34.480850 16305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:34.493394 16305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:34.496079 16305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:34.496234 16305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:34.496465 16305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:23:34.496781 16305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:23:34.496848 16305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:34.496888 16305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:23:34.496914 16305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:34.505198 16305 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:41691
I20260621 05:23:34.505250 16433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:41691 every 8 connection(s)
I20260621 05:23:34.505908 16305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:23:34.509483 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 16305
I20260621 05:23:34.509600 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:23:34.509796 2176 external_mini_cluster.cc:1428] Setting key 50a5d09564af787aea4795fc85e85427
Jun 21 05:23:34 dist-test-slave-b357 krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019414, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:34.520244 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:34.506868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:48777 (local address 127.2.32.62:32835)
0621 05:23:34.506939 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:34.506941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:34.507183 (+ 242us) server_negotiation.cc:407] Connection header received
0621 05:23:34.508222 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:34.508223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:34.508236 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:34.508287 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:34.508902 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.509490 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.511582 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.511950 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.512039 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:34.515031 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:34.515032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:34.515032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:34.515038 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:34.517316 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:34.518782 (+ 1466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:34.518784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:34.518785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:34.518860 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:34.519422 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:34.519423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:34.519424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:34.519571 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:34.519665 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:34.520020 (+ 355us) server_negotiation.cc:299] Negotiation successful
0621 05:23:34.520126 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 05:23:34.520505 16436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:34.506849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.3:48777)
0621 05:23:34.507047 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:34.507081 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:34.508099 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:34.508356 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:34.508360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:34.508621 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:34.508809 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:34.508813 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.509685 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.509688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:34.511454 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:34.511460 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.512066 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.512067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:34.512224 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:34.512916 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:34.512927 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:34.514859 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:34.517986 (+ 3127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:34.517993 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:34.517995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:34.518680 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:34.518981 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:34.518982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:34.518982 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:34.519271 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:34.519677 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:34.519685 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:34.519855 (+ 170us) client_negotiation.cc:769] Sending connection context
0621 05:23:34.519982 (+ 127us) client_negotiation.cc:240] Negotiation successful
0621 05:23:34.520213 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":105,"threads_started":1}
I20260621 05:23:34.521328 16434 heartbeater.cc:347] Connected to a master server at 127.2.32.62:32835
I20260621 05:23:34.521476 16434 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:34.521713 16434 heartbeater.cc:511] Master 127.2.32.62:32835 requested a full tablet report, sending...
I20260621 05:23:34.522205 15974 ts_manager.cc:194] Registered new tserver with Master: 63d927b55814423a9280aae6fdc9083e (127.2.32.3:41691)
I20260621 05:23:34.522639 15974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:48777
I20260621 05:23:34.533288 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:23:34.546205 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:34.534740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.2.32.62:32835)
0621 05:23:34.534823 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:34.534826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:34.534872 (+ 46us) server_negotiation.cc:407] Connection header received
0621 05:23:34.534971 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:34.534972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:34.534984 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:34.535051 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:34.535841 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.536513 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.538878 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.539735 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.539955 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:34.542440 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:34.542442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:34.542443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:34.542452 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:34.544557 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:34.545048 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:34.545050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:34.545051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:34.545126 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:34.545351 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:34.545352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:34.545353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:34.545564 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:34.545693 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:34.545959 (+ 266us) server_negotiation.cc:299] Negotiation successful
0621 05:23:34.546081 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 05:23:34.548635 15974 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:23:34.550254 15974 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:23:34.560844 16450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:34.553800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40142 (local address 127.2.32.1:43139)
0621 05:23:34.554344 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:34.554351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:34.554621 (+ 270us) server_negotiation.cc:407] Connection header received
0621 05:23:34.554669 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:34.554672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:34.554686 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:34.554764 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:23:34.554961 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.556350 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.559270 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.560184 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.560360 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:34.560529 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:34.560682 (+ 153us) server_negotiation.cc:299] Negotiation successful
0621 05:23:34.560741 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":81,"threads_started":1}
I20260621 05:23:34.560840 16449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:34.553666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:43139 (local address 127.0.0.1:40142)
0621 05:23:34.554499 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:34.554508 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:34.554557 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:34.554806 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:34.554807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:34.554811 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:23:34.554864 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:34.554865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.556464 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.556465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:34.558670 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:34.558677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.560365 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.560366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:34.560460 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:34.560479 (+ 19us) client_negotiation.cc:769] Sending connection context
0621 05:23:34.560570 (+ 91us) client_negotiation.cc:240] Negotiation successful
0621 05:23:34.560723 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":17,"thread_start_us":75,"threads_started":1}
I20260621 05:23:34.561393 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:34.553451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:42189 (local address 127.0.0.1:37818)
0621 05:23:34.554059 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:34.554069 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:34.554113 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:34.554407 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:34.554408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:34.554411 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:23:34.554481 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:34.554483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.556243 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.556244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:34.558675 (+ 2431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:34.558677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.561115 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.561116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:34.561187 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:34.561191 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:23:34.561253 (+ 62us) client_negotiation.cc:240] Negotiation successful
0621 05:23:34.561307 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":52,"thread_start_us":99,"threads_started":1}
I20260621 05:23:34.561673 16448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:34.553578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.2.32.2:42189)
0621 05:23:34.554252 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:34.554256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:34.554264 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:23:34.554305 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:34.554307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:34.554318 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:34.554405 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:23:34.554571 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.556126 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.559153 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.560956 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.561024 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:34.561081 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:34.561506 (+ 425us) server_negotiation.cc:299] Negotiation successful
0621 05:23:34.561560 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":102,"threads_started":1}
I20260621 05:23:34.562088 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:34.553324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:41691 (local address 127.0.0.1:54452)
0621 05:23:34.553607 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:34.553658 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:34.553729 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:34.554281 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:34.554288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:34.554292 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:23:34.554428 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:34.554431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.557601 (+ 3170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.557603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:34.560658 (+ 3055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:34.560662 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.561773 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.561774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:34.561848 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:34.561858 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 05:23:34.561960 (+ 102us) client_negotiation.cc:240] Negotiation successful
0621 05:23:34.562007 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":108,"threads_started":1}
I20260621 05:23:34.562132 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:34.553405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54452 (local address 127.2.32.3:41691)
0621 05:23:34.553793 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:34.553799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:34.553810 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:23:34.553854 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:34.553857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:34.553873 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:34.553965 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:23:34.554557 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.556288 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.560799 (+ 4511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.561598 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.561647 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:34.561704 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:34.561990 (+ 286us) server_negotiation.cc:299] Negotiation successful
0621 05:23:34.562038 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":101,"threads_started":1}
I20260621 05:23:34.563625 16368 tablet_service.cc:1467] Processing CreateTablet for tablet 4a119da4257d4201b68086b88cce6a87 (DEFAULT_TABLE table=test-table [id=af045835fde54b87a30504cf5a0677b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:23:34.563679 16100 tablet_service.cc:1467] Processing CreateTablet for tablet 4a119da4257d4201b68086b88cce6a87 (DEFAULT_TABLE table=test-table [id=af045835fde54b87a30504cf5a0677b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:23:34.563638 16234 tablet_service.cc:1467] Processing CreateTablet for tablet 4a119da4257d4201b68086b88cce6a87 (DEFAULT_TABLE table=test-table [id=af045835fde54b87a30504cf5a0677b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:23:34.564563 16100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a119da4257d4201b68086b88cce6a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:23:34.564563 16368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a119da4257d4201b68086b88cce6a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:23:34.564862 16234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a119da4257d4201b68086b88cce6a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:23:34.569453 16451 tablet_bootstrap.cc:492] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33: Bootstrap starting.
I20260621 05:23:34.569736 16452 tablet_bootstrap.cc:492] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e: Bootstrap starting.
I20260621 05:23:34.570240 16453 tablet_bootstrap.cc:492] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e: Bootstrap starting.
I20260621 05:23:34.570479 16452 tablet_bootstrap.cc:654] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e: Neither blocks nor log segments found. Creating new log.
I20260621 05:23:34.570751 16451 tablet_bootstrap.cc:654] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33: Neither blocks nor log segments found. Creating new log.
I20260621 05:23:34.571036 16453 tablet_bootstrap.cc:654] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e: Neither blocks nor log segments found. Creating new log.
I20260621 05:23:34.571096 16452 log.cc:824] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:34.571600 16451 log.cc:824] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:34.571981 16453 log.cc:824] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:34.572605 16452 tablet_bootstrap.cc:492] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e: No bootstrap required, opened a new log
I20260621 05:23:34.572680 16451 tablet_bootstrap.cc:492] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33: No bootstrap required, opened a new log
I20260621 05:23:34.572702 16452 ts_tablet_manager.cc:1397] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:23:34.572757 16451 ts_tablet_manager.cc:1397] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:23:34.572942 16453 tablet_bootstrap.cc:492] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e: No bootstrap required, opened a new log
I20260621 05:23:34.573005 16453 ts_tablet_manager.cc:1397] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:23:34.574993 16451 raft_consensus.cc:348] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:34.575080 16453 raft_consensus.cc:348] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:34.575282 16451 raft_consensus.cc:374] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:23:34.575282 16453 raft_consensus.cc:374] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:23:34.575323 16453 raft_consensus.cc:729] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 956c8c30e25b4adf806e5cd32b30d39e, State: Initialized, Role: FOLLOWER
I20260621 05:23:34.575323 16451 raft_consensus.cc:729] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481ca11e1b404475b9d31696996a6a33, State: Initialized, Role: FOLLOWER
I20260621 05:23:34.575491 16452 raft_consensus.cc:348] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:34.576634 16453 consensus_queue.cc:260] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:34.575903 16451 consensus_queue.cc:260] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:34.576807 16452 raft_consensus.cc:374] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:23:34.576865 16452 raft_consensus.cc:729] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d927b55814423a9280aae6fdc9083e, State: Initialized, Role: FOLLOWER
I20260621 05:23:34.577010 16452 consensus_queue.cc:260] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:34.577567 16166 heartbeater.cc:503] Master 127.2.32.62:32835 was elected leader, sending a full tablet report...
I20260621 05:23:34.577625 16451 ts_tablet_manager.cc:1428] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260621 05:23:34.578249 16434 heartbeater.cc:503] Master 127.2.32.62:32835 was elected leader, sending a full tablet report...
I20260621 05:23:34.578320 16300 heartbeater.cc:503] Master 127.2.32.62:32835 was elected leader, sending a full tablet report...
I20260621 05:23:34.578300 16452 ts_tablet_manager.cc:1428] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260621 05:23:34.579267 16453 ts_tablet_manager.cc:1428] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260621 05:23:34.586450 16459 raft_consensus.cc:482] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:23:34.586561 16459 raft_consensus.cc:504] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:34.586844 16459 leader_election.cc:288] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 956c8c30e25b4adf806e5cd32b30d39e (127.2.32.2:42189), 481ca11e1b404475b9d31696996a6a33 (127.2.32.1:43139)
I20260621 05:23:34.591506 16448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:34.587320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:47131 (local address 127.2.32.2:42189)
0621 05:23:34.587436 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:34.587438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:34.587445 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:23:34.587466 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:34.587467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:34.587478 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:34.587661 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:23:34.587887 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.588655 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.590461 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.591048 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.591147 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:34.591160 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:34.591395 (+ 235us) server_negotiation.cc:299] Negotiation successful
0621 05:23:34.591436 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260621 05:23:34.591506 16450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:34.587200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:37703 (local address 127.2.32.1:43139)
0621 05:23:34.587247 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:34.587249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:34.587631 (+ 382us) server_negotiation.cc:407] Connection header received
0621 05:23:34.587750 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:34.587753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:34.587770 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:34.587812 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:23:34.587976 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.588671 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.590461 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.591048 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.591169 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:34.591179 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:34.591395 (+ 216us) server_negotiation.cc:299] Negotiation successful
0621 05:23:34.591436 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:23:34.591543 16436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:34.586987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:42189 (local address 127.2.32.3:47131)
0621 05:23:34.587056 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:34.587078 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:34.587193 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:34.587705 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:34.587706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:34.587708 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:23:34.587813 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:34.587816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.588777 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.588778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:34.590311 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:34.590314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.591189 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.591190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:34.591285 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:34.591286 (+ 1us) client_negotiation.cc:769] Sending connection context
0621 05:23:34.591357 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 05:23:34.591416 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":105}
I20260621 05:23:34.592006 16460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:34.587082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:43139 (local address 127.2.32.3:37703)
0621 05:23:34.587465 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:34.587474 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:34.587521 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:34.587834 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:34.587835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:34.587837 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:23:34.587903 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:34.587904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.588777 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.588778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:34.590312 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:34.590314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.591189 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.591190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:34.591280 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:34.591285 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:23:34.591357 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 05:23:34.591416 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":99,"thread_start_us":132,"threads_started":1}
I20260621 05:23:34.591914 16254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a119da4257d4201b68086b88cce6a87" candidate_uuid: "63d927b55814423a9280aae6fdc9083e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "956c8c30e25b4adf806e5cd32b30d39e" is_pre_election: true
I20260621 05:23:34.592181 16254 raft_consensus.cc:2454] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d927b55814423a9280aae6fdc9083e in term 0.
I20260621 05:23:34.592344 16120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a119da4257d4201b68086b88cce6a87" candidate_uuid: "63d927b55814423a9280aae6fdc9083e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "481ca11e1b404475b9d31696996a6a33" is_pre_election: true
I20260621 05:23:34.592466 16120 raft_consensus.cc:2454] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d927b55814423a9280aae6fdc9083e in term 0.
I20260621 05:23:34.592665 16321 leader_election.cc:302] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d927b55814423a9280aae6fdc9083e, 956c8c30e25b4adf806e5cd32b30d39e; no voters:
I20260621 05:23:34.592924 16459 raft_consensus.cc:2781] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:23:34.592994 16459 raft_consensus.cc:482] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:23:34.593024 16459 raft_consensus.cc:3037] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:23:34.593750 16459 raft_consensus.cc:504] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:34.593885 16459 leader_election.cc:288] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [CANDIDATE]: Term 1 election: Requested vote from peers 956c8c30e25b4adf806e5cd32b30d39e (127.2.32.2:42189), 481ca11e1b404475b9d31696996a6a33 (127.2.32.1:43139)
I20260621 05:23:34.594084 16254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a119da4257d4201b68086b88cce6a87" candidate_uuid: "63d927b55814423a9280aae6fdc9083e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "956c8c30e25b4adf806e5cd32b30d39e"
I20260621 05:23:34.594183 16254 raft_consensus.cc:3037] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:23:34.594170 16120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a119da4257d4201b68086b88cce6a87" candidate_uuid: "63d927b55814423a9280aae6fdc9083e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "481ca11e1b404475b9d31696996a6a33"
I20260621 05:23:34.594251 16120 raft_consensus.cc:3037] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:23:34.594889 16254 raft_consensus.cc:2454] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d927b55814423a9280aae6fdc9083e in term 1.
I20260621 05:23:34.595036 16120 raft_consensus.cc:2454] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d927b55814423a9280aae6fdc9083e in term 1.
I20260621 05:23:34.595047 16321 leader_election.cc:302] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d927b55814423a9280aae6fdc9083e, 956c8c30e25b4adf806e5cd32b30d39e; no voters:
I20260621 05:23:34.595222 16459 raft_consensus.cc:2781] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:23:34.595933 16459 raft_consensus.cc:686] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [term 1 LEADER]: Becoming Leader. State: Replica: 63d927b55814423a9280aae6fdc9083e, State: Running, Role: LEADER
I20260621 05:23:34.598376 16459 consensus_queue.cc:237] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:34.605168 15973 catalog_manager.cc:5495] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e reported cstate change: term changed from 0 to 1, leader changed from <none> to 63d927b55814423a9280aae6fdc9083e (127.2.32.3). New cstate: current_term: 1 leader_uuid: "63d927b55814423a9280aae6fdc9083e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } health_report { overall_health: UNKNOWN } } }
I20260621 05:23:34.626489 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:34.621851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54458 (local address 127.2.32.3:41691)
0621 05:23:34.621986 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:34.621988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:34.621997 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:23:34.622053 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:34.622054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:34.622080 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:34.622161 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:23:34.622456 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.623210 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.625031 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:34.625356 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:34.625481 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:34.625826 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:23:34.625954 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:23:34.626115 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:34.626339 (+ 224us) server_negotiation.cc:299] Negotiation successful
0621 05:23:34.626388 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260621 05:23:34.629292 16254 raft_consensus.cc:1261] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e [term 1 FOLLOWER]: Refusing update from remote peer 63d927b55814423a9280aae6fdc9083e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:23:34.629295 16120 raft_consensus.cc:1261] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Refusing update from remote peer 63d927b55814423a9280aae6fdc9083e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:23:34.629751 16461 consensus_queue.cc:1035] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:23:34.629914 16459 consensus_queue.cc:1035] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:23:34.632319 16464 mvcc.cc:205] Tried to move back new op lower bound from 7299151522317467648 to 7299151522200776704. Current Snapshot: MvccSnapshot[applied={T|T < 7299151522317467648}]
I20260621 05:23:34.632802 16465 mvcc.cc:205] Tried to move back new op lower bound from 7299151522317467648 to 7299151522200776704. Current Snapshot: MvccSnapshot[applied={T|T < 7299151522317467648}]
I20260621 05:23:34.632946 16467 mvcc.cc:205] Tried to move back new op lower bound from 7299151522317467648 to 7299151522200776704. Current Snapshot: MvccSnapshot[applied={T|T < 7299151522317467648}]
I20260621 05:23:34.635385 15974 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:23:34.635473 15974 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:23:34.636945 15974 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0: Sending DeleteTablet for 3 replicas of tablet 4a119da4257d4201b68086b88cce6a87
I20260621 05:23:34.637377 16234 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a119da4257d4201b68086b88cce6a87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:23:34 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:37818
I20260621 05:23:34.637429 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 16037
I20260621 05:23:34.638710 16368 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a119da4257d4201b68086b88cce6a87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:23:34 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:54452
I20260621 05:23:34.640367 16473 tablet_replica.cc:330] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e: stopping tablet replica
I20260621 05:23:34.641505 16473 raft_consensus.cc:2229] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:23:34.644249 16473 raft_consensus.cc:2258] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:23:34.645128 16474 tablet_replica.cc:330] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e: stopping tablet replica
I20260621 05:23:34.645277 16474 raft_consensus.cc:2229] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:23:34.645264 16473 ts_tablet_manager.cc:1905] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:23:34.645433 16474 raft_consensus.cc:2258] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:23:34.648114 16474 ts_tablet_manager.cc:1905] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:23:34.650084 16474 ts_tablet_manager.cc:1918] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:23:34.650120 16473 ts_tablet_manager.cc:1918] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260621 05:23:34.650126 15959 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.32.1:43139 (error 108)
I20260621 05:23:34.650194 16473 log.cc:1196] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/4a119da4257d4201b68086b88cce6a87
I20260621 05:23:34.650331 16474 log.cc:1196] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/4a119da4257d4201b68086b88cce6a87
I20260621 05:23:34.650475 16473 ts_tablet_manager.cc:1939] T 4a119da4257d4201b68086b88cce6a87 P 956c8c30e25b4adf806e5cd32b30d39e: Deleting consensus metadata
I20260621 05:23:34.650728 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 16171
I20260621 05:23:34.650736 16474 ts_tablet_manager.cc:1939] T 4a119da4257d4201b68086b88cce6a87 P 63d927b55814423a9280aae6fdc9083e: Deleting consensus metadata
I20260621 05:23:34.650964 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:34.650839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:43139 (local address 127.0.0.1:40148)
0621 05:23:34.650892 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:34.650927 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.1:43139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260621 05:23:34.651084 15959 catalog_manager.cc:4584] TS 481ca11e1b404475b9d31696996a6a33 (127.2.32.1:43139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a119da4257d4201b68086b88cce6a87: Network error: Client connection negotiation failed: client connection to 127.2.32.1:43139: connect: Connection refused (error 111)
I20260621 05:23:34.651429 15961 catalog_manager.cc:4841] TS 63d927b55814423a9280aae6fdc9083e (127.2.32.3:41691): tablet 4a119da4257d4201b68086b88cce6a87 (table test-table [id=af045835fde54b87a30504cf5a0677b7]) successfully deleted
I20260621 05:23:34.661862 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 16305
I20260621 05:23:34.662114 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:34.661782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:42189 (local address 127.0.0.1:37828)
0621 05:23:34.661857 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:34.661908 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.2:42189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260621 05:23:34.670446 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 15943
I20260621 05:23:34.677317 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:32835
--webserver_interface=127.2.32.62
--webserver_port=40465
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:32835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:23:34.810992 16476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:34.811311 16476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:34.811358 16476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:34.812830 16476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:23:34.812917 16476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:34.812947 16476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:34.812976 16476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:23:34.813009 16476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:23:34.815223 16476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:32835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:32835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=40465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:34.815534 16476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:34.815804 16476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:34.827461 16485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:34.827842 16483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:34.827848 16482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:34.828130 16476 server_base.cc:1034] running on GCE node
I20260621 05:23:34.828410 16476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:34.828774 16476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:34.829985 16476 hybrid_clock.cc:648] HybridClock initialized: now 1782019414829956 us; error 41 us; skew 500 ppm
Jun 21 05:23:34 dist-test-slave-b357 krb5kdc[15930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019414, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:34.832976 16476 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:23:34.833670 16476 webserver.cc:460] Webserver started at http://127.2.32.62:40465/ using document root <none> and password file <none>
I20260621 05:23:34.834159 16476 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:34.834293 16476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:34.836113 16476 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:34.837304 16492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:34.837683 16476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:34.837833 16476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "e01df39404d24d239c5ac2c66ee4d3e0"
format_stamp: "Formatted at 2026-06-21 05:23:33 on dist-test-slave-b357"
server_key: "0b05095e79aea56f9ed3cd7bd6ad18a9"
server_key_iv: "390f04ce071e7384ced966b02868d6de"
server_key_version: "encryptionkey@0"
I20260621 05:23:34.839748 16476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:34.917176 16476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:34.926806 16476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:34.927031 16476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:34.931964 16476 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:32835
I20260621 05:23:34.931998 16544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:32835 every 8 connection(s)
I20260621 05:23:34.932469 16476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:23:34.934507 16545 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:23:34.935534 16545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0: Bootstrap starting.
I20260621 05:23:34.940253 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 16476
I20260621 05:23:34.940605 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:43139
--local_ip_for_outbound_sockets=127.2.32.1
--tserver_master_addrs=127.2.32.62:32835
--webserver_port=34149
--webserver_interface=127.2.32.1
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:23:34.946492 16545 log.cc:824] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:34.948657 16545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:23:34.948984 16545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0: Bootstrap complete.
I20260621 05:23:34.952132 16545 raft_consensus.cc:348] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 32835 } }
I20260621 05:23:34.952389 16545 raft_consensus.cc:729] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01df39404d24d239c5ac2c66ee4d3e0, State: Initialized, Role: FOLLOWER
I20260621 05:23:34.952593 16545 consensus_queue.cc:260] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 32835 } }
I20260621 05:23:34.952690 16545 raft_consensus.cc:388] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:23:34.952749 16545 raft_consensus.cc:482] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:23:34.952796 16545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 1 FOLLOWER]: Advancing to term 2
I20260621 05:23:34.953980 16545 raft_consensus.cc:504] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 32835 } }
I20260621 05:23:34.954141 16545 leader_election.cc:302] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e01df39404d24d239c5ac2c66ee4d3e0; no voters:
I20260621 05:23:34.954366 16545 leader_election.cc:288] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 05:23:34.954404 16549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 05:23:34.954519 16549 raft_consensus.cc:686] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 2 LEADER]: Becoming Leader. State: Replica: e01df39404d24d239c5ac2c66ee4d3e0, State: Running, Role: LEADER
I20260621 05:23:34.954630 16549 consensus_queue.cc:237] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 32835 } }
I20260621 05:23:34.954663 16545 sys_catalog.cc:564] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:23:34.954907 16549 sys_catalog.cc:455] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e01df39404d24d239c5ac2c66ee4d3e0. Latest consensus state: current_term: 2 leader_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 32835 } } }
I20260621 05:23:34.955003 16549 sys_catalog.cc:458] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [sys.catalog]: This master's current role is: LEADER
I20260621 05:23:34.954936 16550 sys_catalog.cc:455] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 32835 } } }
I20260621 05:23:34.955192 16550 sys_catalog.cc:458] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [sys.catalog]: This master's current role is: LEADER
I20260621 05:23:34.958505 16564 catalog_manager.cc:1259] Loaded cluster ID: bf8cba02be7a4f5d991cb74e7384edd3
I20260621 05:23:34.958565 16564 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0: loading cluster ID for follower catalog manager: success
W20260621 05:23:34.958773 16564 catalog_manager.cc:1576] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260621 05:23:34.958820 16564 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260621 05:23:34.958859 16553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:23:34.960429 16553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:23:34.960592 16553 catalog_manager.cc:1259] Loaded cluster ID: bf8cba02be7a4f5d991cb74e7384edd3
I20260621 05:23:34.960891 16553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:23:34.961817 16553 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260621 05:23:34.966832 16553 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260621 05:23:34.969911 16553 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:23:34.970224 16553 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0: Loaded TSK: 0
I20260621 05:23:34.970476 16553 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 05:23:35.052412 16547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:35.052713 16547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:35.052757 16547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:35.055423 16547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:35.055514 16547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:35.055570 16547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:23:35.060242 16547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:43139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=34149
--tserver_master_addrs=127.2.32.62:32835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:35.060672 16547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:35.060987 16547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:35.067514 16573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:35.067507 16572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:35.067677 16547 server_base.cc:1034] running on GCE node
W20260621 05:23:35.067500 16575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:35.068369 16547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:35.068732 16547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:35.069928 16547 hybrid_clock.cc:648] HybridClock initialized: now 1782019415069891 us; error 51 us; skew 500 ppm
Jun 21 05:23:35 dist-test-slave-b357 krb5kdc[15930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019415, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:35.072556 16547 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:23:35.073062 16547 webserver.cc:460] Webserver started at http://127.2.32.1:34149/ using document root <none> and password file <none>
I20260621 05:23:35.073311 16547 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:35.073369 16547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:35.074749 16547 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:23:35.076118 16582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:35.076800 16547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:23:35.076998 16547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "481ca11e1b404475b9d31696996a6a33"
format_stamp: "Formatted at 2026-06-21 05:23:34 on dist-test-slave-b357"
server_key: "bdcd702c76863608cd79b2928a66363c"
server_key_iv: "c1037bba371917cf42f6f25b3246c07f"
server_key_version: "encryptionkey@0"
I20260621 05:23:35.077478 16547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:35.095573 16547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:35.098614 16547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:35.099272 16547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:35.099757 16547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:23:35.100531 16589 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 05:23:35.102499 16547 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 05:23:35.102555 16547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260621 05:23:35.102583 16547 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 05:23:35.103320 16547 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 05:23:35.103369 16547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:35.103400 16589 tablet_bootstrap.cc:492] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33: Bootstrap starting.
I20260621 05:23:35.112326 16547 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:43139
I20260621 05:23:35.112707 16696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:43139 every 8 connection(s)
I20260621 05:23:35.112846 16547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:23:35.117061 16589 log.cc:824] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:35.119786 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 16547
I20260621 05:23:35.120146 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:42189
--local_ip_for_outbound_sockets=127.2.32.2
--tserver_master_addrs=127.2.32.62:32835
--webserver_port=40425
--webserver_interface=127.2.32.2
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:23:35.120836 16589 tablet_bootstrap.cc:492] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:23:35.122339 16589 tablet_bootstrap.cc:492] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33: Bootstrap complete.
I20260621 05:23:35.123481 16589 ts_tablet_manager.cc:1397] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.015s
Jun 21 05:23:35 dist-test-slave-b357 krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019415, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:35.124858 16589 raft_consensus.cc:348] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:35.125232 16589 raft_consensus.cc:729] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481ca11e1b404475b9d31696996a6a33, State: Initialized, Role: FOLLOWER
I20260621 05:23:35.125412 16589 consensus_queue.cc:260] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:35.125717 16589 ts_tablet_manager.cc:1428] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 05:23:35.128671 16699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:35.113848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.1:48099)
0621 05:23:35.114027 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:35.114069 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:35.114809 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:35.115811 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:35.115817 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:35.116275 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:35.116547 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:35.116554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.118505 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.118507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:35.119955 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:35.119960 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.121200 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.121201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:35.121353 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:35.121974 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:35.121985 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:35.124174 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:35.126725 (+ 2551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:35.126732 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:35.126734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:35.127054 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:35.127395 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:35.127396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:35.127396 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:35.127535 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:35.127997 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:35.128003 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:35.128151 (+ 148us) client_negotiation.cc:769] Sending connection context
0621 05:23:35.128235 (+ 84us) client_negotiation.cc:240] Negotiation successful
0621 05:23:35.128417 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":104,"threads_started":1}
I20260621 05:23:35.129312 16700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:35.113955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:48099 (local address 127.2.32.62:32835)
0621 05:23:35.115375 (+ 1420us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:35.115385 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:35.115422 (+ 37us) server_negotiation.cc:407] Connection header received
0621 05:23:35.115550 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:35.115556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:35.115606 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:35.115750 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:35.116963 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.118320 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.120167 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.121064 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.121631 (+ 567us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:35.124355 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:35.124359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:35.124360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:35.124371 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:35.126618 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:35.127179 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:35.127182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:35.127182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:35.127294 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:35.127611 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:35.127612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:35.127613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:35.127915 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:35.128848 (+ 933us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:35.128927 (+ 79us) server_negotiation.cc:299] Negotiation successful
0621 05:23:35.129080 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1370,"thread_start_us":88,"threads_started":1}
I20260621 05:23:35.129899 16697 heartbeater.cc:347] Connected to a master server at 127.2.32.62:32835
I20260621 05:23:35.129998 16697 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:35.130172 16697 heartbeater.cc:511] Master 127.2.32.62:32835 requested a full tablet report, sending...
I20260621 05:23:35.130914 16509 ts_manager.cc:194] Registered new tserver with Master: 481ca11e1b404475b9d31696996a6a33 (127.2.32.1:43139)
I20260621 05:23:35.132231 16509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:48099
I20260621 05:23:35.133179 16697 heartbeater.cc:503] Master 127.2.32.62:32835 was elected leader, sending a full tablet report...
I20260621 05:23:35.138103 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:35.131602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:43139 (local address 127.0.0.1:40158)
0621 05:23:35.134167 (+ 2565us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:35.134215 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:35.134316 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:35.134555 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:35.134558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:35.134560 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:23:35.134639 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:35.134641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.135765 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.135767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:35.136715 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:35.136719 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.137769 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.137770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:35.137853 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:35.137865 (+ 12us) client_negotiation.cc:769] Sending connection context
0621 05:23:35.137970 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 05:23:35.138028 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2518,"thread_start_us":79,"threads_started":1}
I20260621 05:23:35.138103 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:35.131835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40158 (local address 127.2.32.1:43139)
0621 05:23:35.131993 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:35.131998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:35.134293 (+ 2295us) server_negotiation.cc:407] Connection header received
0621 05:23:35.134423 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:35.134426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:35.134444 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:35.134551 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:23:35.134728 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.135533 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.136840 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.137569 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.137767 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:35.137825 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:35.137988 (+ 163us) server_negotiation.cc:299] Negotiation successful
0621 05:23:35.138028 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260621 05:23:35.139041 16631 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a119da4257d4201b68086b88cce6a87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:23:34 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:40158
I20260621 05:23:35.139887 16706 tablet_replica.cc:330] stopping tablet replica
I20260621 05:23:35.140043 16706 raft_consensus.cc:2229] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:23:35.140125 16706 raft_consensus.cc:2258] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:23:35.140585 16706 ts_tablet_manager.cc:1905] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:23:35.143297 16706 ts_tablet_manager.cc:1918] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:23:35.143385 16706 log.cc:1196] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/wal/wals/4a119da4257d4201b68086b88cce6a87
I20260621 05:23:35.143632 16706 ts_tablet_manager.cc:1939] T 4a119da4257d4201b68086b88cce6a87 P 481ca11e1b404475b9d31696996a6a33: Deleting consensus metadata
I20260621 05:23:35.144371 16493 catalog_manager.cc:4841] TS 481ca11e1b404475b9d31696996a6a33 (127.2.32.1:43139): tablet 4a119da4257d4201b68086b88cce6a87 (table test-table [id=af045835fde54b87a30504cf5a0677b7]) successfully deleted
W20260621 05:23:35.214681 16702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:35.214875 16702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:35.214897 16702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:35.216064 16702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:35.216109 16702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:35.216142 16702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:23:35.217525 16702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:42189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=40425
--tserver_master_addrs=127.2.32.62:32835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:35.217722 16702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:35.217929 16702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:35.224542 16711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:35.224785 16702 server_base.cc:1034] running on GCE node
W20260621 05:23:35.224440 16714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:35.224448 16712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:35.225193 16702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:35.225556 16702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:35.226742 16702 hybrid_clock.cc:648] HybridClock initialized: now 1782019415226728 us; error 37 us; skew 500 ppm
Jun 21 05:23:35 dist-test-slave-b357 krb5kdc[15930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019415, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:35.229658 16702 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:23:35.230305 16702 webserver.cc:460] Webserver started at http://127.2.32.2:40425/ using document root <none> and password file <none>
I20260621 05:23:35.230672 16702 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:35.230794 16702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:35.232510 16702 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:35.233264 16721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:35.233544 16702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:35.233641 16702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "956c8c30e25b4adf806e5cd32b30d39e"
format_stamp: "Formatted at 2026-06-21 05:23:34 on dist-test-slave-b357"
server_key: "2e6e1cbb9d0785b396ba899838bed749"
server_key_iv: "4e8d576fe664a6c282cd5934e8388af9"
server_key_version: "encryptionkey@0"
I20260621 05:23:35.234017 16702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:35.256201 16702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:35.259019 16702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:35.259217 16702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:35.259480 16702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:23:35.259809 16702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:23:35.259867 16702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:35.259909 16702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:23:35.259938 16702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:35.267717 16702 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:42189
I20260621 05:23:35.267804 16834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:42189 every 8 connection(s)
I20260621 05:23:35.268147 16702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:23:35.271289 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 16702
I20260621 05:23:35.271669 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:41691
--local_ip_for_outbound_sockets=127.2.32.3
--tserver_master_addrs=127.2.32.62:32835
--webserver_port=45583
--webserver_interface=127.2.32.3
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 05:23:35 dist-test-slave-b357 krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019415, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:35.282366 16700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:35.269032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:44681 (local address 127.2.32.62:32835)
0621 05:23:35.269145 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:35.269147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:35.269311 (+ 164us) server_negotiation.cc:407] Connection header received
0621 05:23:35.269802 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:35.269804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:35.269813 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:35.269876 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:35.271577 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.272221 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.274075 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.274401 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.274488 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:35.277909 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:35.277911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:35.277920 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:35.277930 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:35.279793 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:35.280517 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:35.280518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:35.280519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:35.280592 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:35.281296 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:35.281297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:35.281297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:35.281484 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:35.281597 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:35.282167 (+ 570us) server_negotiation.cc:299] Negotiation successful
0621 05:23:35.282284 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 05:23:35.282497 16837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:35.268952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.2:44681)
0621 05:23:35.269199 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:35.269229 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:35.269674 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:35.270172 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:35.270178 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:35.270928 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:35.271404 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:35.271415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.272342 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.272343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:35.273856 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:35.273860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.274533 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.274535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:35.274664 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:35.275550 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:35.275560 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:35.277599 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:35.279904 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:35.279911 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:35.279914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:35.280260 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:35.280712 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:35.280713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:35.280713 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:35.281097 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:35.281592 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:35.281599 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:35.281808 (+ 209us) client_negotiation.cc:769] Sending connection context
0621 05:23:35.281995 (+ 187us) client_negotiation.cc:240] Negotiation successful
0621 05:23:35.282199 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":85,"threads_started":1}
I20260621 05:23:35.283015 16835 heartbeater.cc:347] Connected to a master server at 127.2.32.62:32835
I20260621 05:23:35.283145 16835 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:35.283357 16835 heartbeater.cc:511] Master 127.2.32.62:32835 requested a full tablet report, sending...
I20260621 05:23:35.283879 16509 ts_manager.cc:194] Registered new tserver with Master: 956c8c30e25b4adf806e5cd32b30d39e (127.2.32.2:42189)
I20260621 05:23:35.284312 16509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:44681
W20260621 05:23:35.373633 16838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:35.373912 16838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:35.373940 16838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:35.375349 16838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:35.375406 16838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:35.375437 16838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:23:35.377794 16838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:41691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=45583
--tserver_master_addrs=127.2.32.62:32835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:35.378239 16838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:35.378649 16838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:23:35.384877 16838 server_base.cc:1034] running on GCE node
W20260621 05:23:35.384837 16844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:35.384837 16843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:35.384850 16846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:35.385393 16838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:35.385668 16838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:35.386863 16838 hybrid_clock.cc:648] HybridClock initialized: now 1782019415386814 us; error 68 us; skew 500 ppm
Jun 21 05:23:35 dist-test-slave-b357 krb5kdc[15930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019415, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:35.390048 16838 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:23:35.390664 16838 webserver.cc:460] Webserver started at http://127.2.32.3:45583/ using document root <none> and password file <none>
I20260621 05:23:35.390925 16838 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:35.391008 16838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:35.392652 16838 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:35.393410 16853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:35.393743 16838 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:23:35.393844 16838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "63d927b55814423a9280aae6fdc9083e"
format_stamp: "Formatted at 2026-06-21 05:23:34 on dist-test-slave-b357"
server_key: "7a8ffabf4e855250c06dbfd6afc27e0d"
server_key_iv: "bccdb0bcf4c4defd0888e04f67b541e9"
server_key_version: "encryptionkey@0"
I20260621 05:23:35.394210 16838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:35.421834 16838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:35.424197 16838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:35.424371 16838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:35.424630 16838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:23:35.424947 16838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:23:35.425004 16838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:35.425046 16838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:23:35.425081 16838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:35.431466 16838 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:41691
I20260621 05:23:35.431509 16966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:41691 every 8 connection(s)
I20260621 05:23:35.431838 16838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:23:35 dist-test-slave-b357 krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019415, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:35.439751 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 16838
I20260621 05:23:35.443611 16700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:35.432527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:58061 (local address 127.2.32.62:32835)
0621 05:23:35.432654 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:35.432656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:35.432819 (+ 163us) server_negotiation.cc:407] Connection header received
0621 05:23:35.433327 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:35.433330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:35.433340 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:35.433400 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:35.434434 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.435000 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.436239 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.436511 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.436612 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:35.439649 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:35.439651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:35.439651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:35.439659 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:35.441844 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:35.442328 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:35.442329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:35.442330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:35.442453 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:35.442728 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:35.442729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:35.442729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:35.442864 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:35.443008 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:35.443409 (+ 401us) server_negotiation.cc:299] Negotiation successful
0621 05:23:35.443504 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260621 05:23:35.443836 16969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:35.432508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.3:58061)
0621 05:23:35.432696 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:35.432737 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:35.433243 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:35.433469 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:35.433472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:35.434131 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:35.434312 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:35.434317 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.435097 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.435098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:35.435961 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:35.435964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.436651 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.436652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:35.436790 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:35.437418 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:35.437426 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:35.439471 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:35.441956 (+ 2485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:35.441967 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:35.441970 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:35.442244 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:35.442538 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:35.442538 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:35.442539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:35.442658 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:35.443000 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:35.443007 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:35.443222 (+ 215us) client_negotiation.cc:769] Sending connection context
0621 05:23:35.443352 (+ 130us) client_negotiation.cc:240] Negotiation successful
0621 05:23:35.443562 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":118,"threads_started":1}
I20260621 05:23:35.444834 16967 heartbeater.cc:347] Connected to a master server at 127.2.32.62:32835
I20260621 05:23:35.444928 16967 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:35.445125 16967 heartbeater.cc:511] Master 127.2.32.62:32835 requested a full tablet report, sending...
I20260621 05:23:35.445569 16509 ts_manager.cc:194] Registered new tserver with Master: 63d927b55814423a9280aae6fdc9083e (127.2.32.3:41691)
I20260621 05:23:35.445943 16509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:58061
I20260621 05:23:35.448534 16971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:35.440461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60664 (local address 127.2.32.62:32835)
0621 05:23:35.440579 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:35.440581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:35.440653 (+ 72us) server_negotiation.cc:407] Connection header received
0621 05:23:35.440767 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:35.440767 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:35.440779 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:35.440915 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:35.441857 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.442459 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.443426 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.443666 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.443731 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:35.445420 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:35.445421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:35.445422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:35.445429 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:35.446999 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:35.447429 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:35.447430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:35.447430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:35.447481 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:35.447849 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:35.447850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:35.447850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:35.447955 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:35.448086 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:35.448316 (+ 230us) server_negotiation.cc:299] Negotiation successful
0621 05:23:35.448388 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
I20260621 05:23:35.449517 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:23:35.452257 16971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:35.449889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.2.32.62:32835)
0621 05:23:35.449944 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:35.449946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:35.450024 (+ 78us) server_negotiation.cc:407] Connection header received
0621 05:23:35.450082 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:35.450083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:35.450104 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:35.450160 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:23:35.450269 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.450695 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.451483 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.451692 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.451748 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:35.451944 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:23:35.452033 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:23:35.452112 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:35.452198 (+ 86us) server_negotiation.cc:299] Negotiation successful
0621 05:23:35.452215 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 05:23:35.452505 16509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:23:35.454934 16509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:23:35.457734 16631 tablet_service.cc:1467] Processing CreateTablet for tablet 1f911dfb1db946ff9c222375339c883c (DEFAULT_TABLE table=test-table [id=e7529891828c4ecfa12fc4d10a42806c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:23:35.458010 16631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f911dfb1db946ff9c222375339c883c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:23:35.459813 16589 tablet_bootstrap.cc:492] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33: Bootstrap starting.
I20260621 05:23:35.460196 16589 tablet_bootstrap.cc:654] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33: Neither blocks nor log segments found. Creating new log.
I20260621 05:23:35.460734 16589 tablet_bootstrap.cc:492] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33: No bootstrap required, opened a new log
I20260621 05:23:35.460798 16589 ts_tablet_manager.cc:1397] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260621 05:23:35.460934 16589 raft_consensus.cc:348] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:35.461016 16589 raft_consensus.cc:374] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:23:35.461071 16589 raft_consensus.cc:729] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481ca11e1b404475b9d31696996a6a33, State: Initialized, Role: FOLLOWER
I20260621 05:23:35.461131 16589 consensus_queue.cc:260] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:35.461241 16589 ts_tablet_manager.cc:1428] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260621 05:23:35.462049 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:35.457190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:41691 (local address 127.0.0.1:54472)
0621 05:23:35.457544 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:35.457555 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:35.457601 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:35.458182 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:35.458183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:35.458185 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:23:35.458224 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:35.458225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.459031 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.459032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:35.460516 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:35.460520 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.461751 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.461753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:35.461831 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:35.461836 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:23:35.461896 (+ 60us) client_negotiation.cc:240] Negotiation successful
0621 05:23:35.461973 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":25,"thread_start_us":77,"threads_started":1}
I20260621 05:23:35.462184 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:35.457190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:42189 (local address 127.0.0.1:37830)
0621 05:23:35.457420 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:35.457433 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:35.457480 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:35.457827 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:35.457828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:35.457829 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:23:35.457864 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:35.457865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.459287 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.459287 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:35.460604 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:35.460606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.461887 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.461888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:35.461953 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:35.461958 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:23:35.461992 (+ 34us) client_negotiation.cc:240] Negotiation successful
0621 05:23:35.462056 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
I20260621 05:23:35.462386 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:35.457325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37830 (local address 127.2.32.2:42189)
0621 05:23:35.457594 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:35.457599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:35.457613 (+ 14us) server_negotiation.cc:407] Connection header received
0621 05:23:35.457663 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:35.457665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:35.457684 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:35.457823 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:23:35.458012 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.459111 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.460767 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.461784 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.462133 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:35.462205 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:35.462235 (+ 30us) server_negotiation.cc:299] Negotiation successful
0621 05:23:35.462283 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":118,"threads_started":1}
I20260621 05:23:35.462505 16975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:35.457565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54472 (local address 127.2.32.3:41691)
0621 05:23:35.457953 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:35.457958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:35.457967 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:23:35.458005 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:35.458007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:35.458046 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:35.458156 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:23:35.458278 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.458931 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.460682 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.461643 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.461747 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:35.461814 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:35.462262 (+ 448us) server_negotiation.cc:299] Negotiation successful
0621 05:23:35.462315 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":91,"threads_started":1}
I20260621 05:23:35.463115 16769 tablet_service.cc:1467] Processing CreateTablet for tablet 1f911dfb1db946ff9c222375339c883c (DEFAULT_TABLE table=test-table [id=e7529891828c4ecfa12fc4d10a42806c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:23:35.463202 16901 tablet_service.cc:1467] Processing CreateTablet for tablet 1f911dfb1db946ff9c222375339c883c (DEFAULT_TABLE table=test-table [id=e7529891828c4ecfa12fc4d10a42806c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:23:35.463531 16901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f911dfb1db946ff9c222375339c883c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:23:35.463531 16769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f911dfb1db946ff9c222375339c883c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:23:35.466239 16977 tablet_bootstrap.cc:492] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e: Bootstrap starting.
I20260621 05:23:35.466882 16977 tablet_bootstrap.cc:654] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e: Neither blocks nor log segments found. Creating new log.
I20260621 05:23:35.467049 16978 tablet_bootstrap.cc:492] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e: Bootstrap starting.
I20260621 05:23:35.467492 16977 log.cc:824] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:35.467829 16978 tablet_bootstrap.cc:654] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e: Neither blocks nor log segments found. Creating new log.
I20260621 05:23:35.468485 16977 tablet_bootstrap.cc:492] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e: No bootstrap required, opened a new log
I20260621 05:23:35.468550 16978 log.cc:824] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:35.468628 16977 ts_tablet_manager.cc:1397] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:23:35.469421 16978 tablet_bootstrap.cc:492] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e: No bootstrap required, opened a new log
I20260621 05:23:35.469508 16978 ts_tablet_manager.cc:1397] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260621 05:23:35.470575 16977 raft_consensus.cc:348] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:35.470754 16977 raft_consensus.cc:374] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:23:35.470793 16977 raft_consensus.cc:729] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d927b55814423a9280aae6fdc9083e, State: Initialized, Role: FOLLOWER
I20260621 05:23:35.470906 16977 consensus_queue.cc:260] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:35.471236 16977 ts_tablet_manager.cc:1428] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:23:35.471364 16967 heartbeater.cc:503] Master 127.2.32.62:32835 was elected leader, sending a full tablet report...
I20260621 05:23:35.471921 16978 raft_consensus.cc:348] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:35.472103 16978 raft_consensus.cc:374] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:23:35.472146 16978 raft_consensus.cc:729] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 956c8c30e25b4adf806e5cd32b30d39e, State: Initialized, Role: FOLLOWER
I20260621 05:23:35.472258 16978 consensus_queue.cc:260] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:35.472517 16978 ts_tablet_manager.cc:1428] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 05:23:35.472575 16835 heartbeater.cc:503] Master 127.2.32.62:32835 was elected leader, sending a full tablet report...
W20260621 05:23:35.518777 16836 tablet.cc:2369] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:23:35.610044 16982 raft_consensus.cc:482] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:23:35.610231 16982 raft_consensus.cc:504] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:35.610634 16982 leader_election.cc:288] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63d927b55814423a9280aae6fdc9083e (127.2.32.3:41691), 481ca11e1b404475b9d31696996a6a33 (127.2.32.1:43139)
W20260621 05:23:35.613972 16698 tablet.cc:2369] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:23:35.614332 16837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:35.610879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:43139 (local address 127.2.32.2:48197)
0621 05:23:35.610964 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:35.610981 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:35.611043 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:35.611310 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:35.611311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:35.611313 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:23:35.611434 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:35.611438 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.612207 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.612208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:35.613457 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:35.613461 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.614060 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.614061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:35.614155 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:35.614159 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:23:35.614208 (+ 49us) client_negotiation.cc:240] Negotiation successful
0621 05:23:35.614259 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 05:23:35.614333 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:35.610985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:48197 (local address 127.2.32.1:43139)
0621 05:23:35.611050 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:35.611052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:35.611100 (+ 48us) server_negotiation.cc:407] Connection header received
0621 05:23:35.611160 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:35.611161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:35.611172 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:35.611235 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:23:35.611530 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.612106 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.613600 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.613949 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.614049 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:35.614060 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:35.614231 (+ 171us) server_negotiation.cc:299] Negotiation successful
0621 05:23:35.614272 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 05:23:35.614607 16975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:35.611171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:37939 (local address 127.2.32.3:41691)
0621 05:23:35.611242 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:35.611245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:35.611448 (+ 203us) server_negotiation.cc:407] Connection header received
0621 05:23:35.611484 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:35.611485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:35.611503 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:35.611571 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:23:35.611807 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.612465 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.613491 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.614042 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.614120 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:35.614134 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:35.614343 (+ 209us) server_negotiation.cc:299] Negotiation successful
0621 05:23:35.614392 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 05:23:35.614635 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:35.611099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:41691 (local address 127.2.32.2:37939)
0621 05:23:35.611359 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:35.611371 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:35.611427 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:35.611591 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:35.611592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:35.611594 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:23:35.611678 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:35.611680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.612583 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.612584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:35.613367 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:35.613369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.614176 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.614177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:35.614272 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:35.614278 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:23:35.614328 (+ 50us) client_negotiation.cc:240] Negotiation successful
0621 05:23:35.614382 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":170,"threads_started":1}
I20260621 05:23:35.614648 16647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f911dfb1db946ff9c222375339c883c" candidate_uuid: "956c8c30e25b4adf806e5cd32b30d39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "481ca11e1b404475b9d31696996a6a33" is_pre_election: true
I20260621 05:23:35.614825 16647 raft_consensus.cc:2454] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 956c8c30e25b4adf806e5cd32b30d39e in term 0.
I20260621 05:23:35.615056 16921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f911dfb1db946ff9c222375339c883c" candidate_uuid: "956c8c30e25b4adf806e5cd32b30d39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d927b55814423a9280aae6fdc9083e" is_pre_election: true
I20260621 05:23:35.615082 16722 leader_election.cc:302] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 481ca11e1b404475b9d31696996a6a33, 956c8c30e25b4adf806e5cd32b30d39e; no voters:
I20260621 05:23:35.615226 16921 raft_consensus.cc:2454] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 956c8c30e25b4adf806e5cd32b30d39e in term 0.
I20260621 05:23:35.615247 16982 raft_consensus.cc:2781] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:23:35.615298 16982 raft_consensus.cc:482] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:23:35.615324 16982 raft_consensus.cc:3037] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:23:35.616083 16982 raft_consensus.cc:504] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:35.616235 16982 leader_election.cc:288] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [CANDIDATE]: Term 1 election: Requested vote from peers 63d927b55814423a9280aae6fdc9083e (127.2.32.3:41691), 481ca11e1b404475b9d31696996a6a33 (127.2.32.1:43139)
I20260621 05:23:35.616429 16921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f911dfb1db946ff9c222375339c883c" candidate_uuid: "956c8c30e25b4adf806e5cd32b30d39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d927b55814423a9280aae6fdc9083e"
I20260621 05:23:35.616456 16647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f911dfb1db946ff9c222375339c883c" candidate_uuid: "956c8c30e25b4adf806e5cd32b30d39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "481ca11e1b404475b9d31696996a6a33"
I20260621 05:23:35.616528 16921 raft_consensus.cc:3037] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:23:35.616528 16647 raft_consensus.cc:3037] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:23:35.617449 16921 raft_consensus.cc:2454] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 956c8c30e25b4adf806e5cd32b30d39e in term 1.
I20260621 05:23:35.617744 16724 leader_election.cc:302] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d927b55814423a9280aae6fdc9083e, 956c8c30e25b4adf806e5cd32b30d39e; no voters:
I20260621 05:23:35.617866 16982 raft_consensus.cc:2781] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:23:35.618017 16647 raft_consensus.cc:2454] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 956c8c30e25b4adf806e5cd32b30d39e in term 1.
I20260621 05:23:35.618372 16982 raft_consensus.cc:686] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [term 1 LEADER]: Becoming Leader. State: Replica: 956c8c30e25b4adf806e5cd32b30d39e, State: Running, Role: LEADER
I20260621 05:23:35.618566 16982 consensus_queue.cc:237] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:35.619764 16509 catalog_manager.cc:5495] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e reported cstate change: term changed from 0 to 1, leader changed from <none> to 956c8c30e25b4adf806e5cd32b30d39e (127.2.32.2). New cstate: current_term: 1 leader_uuid: "956c8c30e25b4adf806e5cd32b30d39e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } health_report { overall_health: UNKNOWN } } }
I20260621 05:23:35.648039 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:35.644982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.2.32.2:42189)
0621 05:23:35.645064 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:35.645066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:35.645083 (+ 17us) server_negotiation.cc:407] Connection header received
0621 05:23:35.645125 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:35.645125 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:35.645149 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:35.645217 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:23:35.645629 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.646105 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.647191 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:35.647413 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:35.647483 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:35.647663 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:23:35.647767 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:23:35.647843 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:35.647915 (+ 72us) server_negotiation.cc:299] Negotiation successful
0621 05:23:35.647935 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 05:23:35.650134 16647 raft_consensus.cc:1261] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Refusing update from remote peer 956c8c30e25b4adf806e5cd32b30d39e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:23:35.650134 16921 raft_consensus.cc:1261] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e [term 1 FOLLOWER]: Refusing update from remote peer 956c8c30e25b4adf806e5cd32b30d39e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:23:35.650667 16982 consensus_queue.cc:1035] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:23:35.650821 16984 consensus_queue.cc:1035] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:23:35.652827 16988 mvcc.cc:205] Tried to move back new op lower bound from 7299151526499115008 to 7299151526374436864. Current Snapshot: MvccSnapshot[applied={T|T < 7299151526499115008}]
I20260621 05:23:35.655337 16508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:23:35.655462 16508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:23:35.656431 16508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0: Sending DeleteTablet for 3 replicas of tablet 1f911dfb1db946ff9c222375339c883c
I20260621 05:23:35.656842 16769 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f911dfb1db946ff9c222375339c883c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:23:35 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:37830
I20260621 05:23:35.656968 16631 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f911dfb1db946ff9c222375339c883c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:23:35 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:40158
I20260621 05:23:35.656931 16901 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f911dfb1db946ff9c222375339c883c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:23:35 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:54472
I20260621 05:23:35.657267 16999 tablet_replica.cc:330] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33: stopping tablet replica
I20260621 05:23:35.657341 16999 raft_consensus.cc:2229] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:23:35.657384 16999 raft_consensus.cc:2258] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:23:35.657461 16997 tablet_replica.cc:330] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e: stopping tablet replica
I20260621 05:23:35.657495 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 16547
I20260621 05:23:35.657522 16998 tablet_replica.cc:330] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e: stopping tablet replica
I20260621 05:23:35.657614 16997 raft_consensus.cc:2229] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:23:35.657626 16998 raft_consensus.cc:2229] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:23:35.657732 16998 raft_consensus.cc:2258] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:23:35.657750 16997 raft_consensus.cc:2258] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:23:35.658480 16998 ts_tablet_manager.cc:1905] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:23:35.661121 16998 ts_tablet_manager.cc:1918] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:23:35.661227 16998 log.cc:1196] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/1f911dfb1db946ff9c222375339c883c
I20260621 05:23:35.661583 16998 ts_tablet_manager.cc:1939] T 1f911dfb1db946ff9c222375339c883c P 63d927b55814423a9280aae6fdc9083e: Deleting consensus metadata
I20260621 05:23:35.662321 16495 catalog_manager.cc:4841] TS 63d927b55814423a9280aae6fdc9083e (127.2.32.3:41691): tablet 1f911dfb1db946ff9c222375339c883c (table test-table [id=e7529891828c4ecfa12fc4d10a42806c]) successfully deleted
I20260621 05:23:35.662665 16997 ts_tablet_manager.cc:1905] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:23:35.665000 16997 ts_tablet_manager.cc:1918] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:23:35.665107 16997 log.cc:1196] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/1f911dfb1db946ff9c222375339c883c
I20260621 05:23:35.665412 16997 ts_tablet_manager.cc:1939] T 1f911dfb1db946ff9c222375339c883c P 956c8c30e25b4adf806e5cd32b30d39e: Deleting consensus metadata
I20260621 05:23:35.665911 16493 catalog_manager.cc:4841] TS 956c8c30e25b4adf806e5cd32b30d39e (127.2.32.2:42189): tablet 1f911dfb1db946ff9c222375339c883c (table test-table [id=e7529891828c4ecfa12fc4d10a42806c]) successfully deleted
W20260621 05:23:35.666930 16493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.32.1:43139 (error 108)
I20260621 05:23:35.667531 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:35.667395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:43139 (local address 127.0.0.1:40160)
0621 05:23:35.667458 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:35.667497 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.1:43139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260621 05:23:35.667605 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 16702
W20260621 05:23:35.667659 16493 catalog_manager.cc:4584] TS 481ca11e1b404475b9d31696996a6a33 (127.2.32.1:43139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f911dfb1db946ff9c222375339c883c: Network error: Client connection negotiation failed: client connection to 127.2.32.1:43139: connect: Connection refused (error 111)
I20260621 05:23:35.673516 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 16838
I20260621 05:23:35.678056 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 16476
I20260621 05:23:35.683809 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:32835
--webserver_interface=127.2.32.62
--webserver_port=40465
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:32835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:23:35.772405 17001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:35.772605 17001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:35.772627 17001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:35.773746 17001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:23:35.773787 17001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:35.773801 17001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:35.773813 17001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:23:35.773825 17001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:23:35.775315 17001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:32835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:32835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=40465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:35.775554 17001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:35.775789 17001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:35.781273 17007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:35.781275 17009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:35.781273 17006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:35.781728 17001 server_base.cc:1034] running on GCE node
I20260621 05:23:35.782035 17001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:35.782310 17001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:35.783672 17001 hybrid_clock.cc:648] HybridClock initialized: now 1782019415783641 us; error 46 us; skew 500 ppm
Jun 21 05:23:35 dist-test-slave-b357 krb5kdc[15930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019415, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:35.786824 17001 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:23:35.787464 17001 webserver.cc:460] Webserver started at http://127.2.32.62:40465/ using document root <none> and password file <none>
I20260621 05:23:35.787704 17001 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:35.787775 17001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:35.789266 17001 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:35.790135 17016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:35.790441 17001 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:35.790568 17001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "e01df39404d24d239c5ac2c66ee4d3e0"
format_stamp: "Formatted at 2026-06-21 05:23:33 on dist-test-slave-b357"
server_key: "0b05095e79aea56f9ed3cd7bd6ad18a9"
server_key_iv: "390f04ce071e7384ced966b02868d6de"
server_key_version: "encryptionkey@0"
I20260621 05:23:35.790917 17001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:35.803954 17001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:35.806548 17001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:35.806740 17001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:35.811756 17001 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:32835
I20260621 05:23:35.811829 17068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:32835 every 8 connection(s)
I20260621 05:23:35.812194 17001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:23:35.813771 17069 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:23:35.814424 17069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0: Bootstrap starting.
I20260621 05:23:35.820112 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 17001
I20260621 05:23:35.820557 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:43139
--local_ip_for_outbound_sockets=127.2.32.1
--tserver_master_addrs=127.2.32.62:32835
--webserver_port=34149
--webserver_interface=127.2.32.1
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:23:35.826746 17069 log.cc:824] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:35.829099 17069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:23:35.829433 17069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0: Bootstrap complete.
I20260621 05:23:35.831849 17069 raft_consensus.cc:348] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 32835 } }
I20260621 05:23:35.832072 17069 raft_consensus.cc:729] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01df39404d24d239c5ac2c66ee4d3e0, State: Initialized, Role: FOLLOWER
I20260621 05:23:35.832247 17069 consensus_queue.cc:260] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 32835 } }
I20260621 05:23:35.832307 17069 raft_consensus.cc:388] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:23:35.832363 17069 raft_consensus.cc:482] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:23:35.832412 17069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 2 FOLLOWER]: Advancing to term 3
I20260621 05:23:35.833411 17069 raft_consensus.cc:504] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 32835 } }
I20260621 05:23:35.833539 17069 leader_election.cc:302] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e01df39404d24d239c5ac2c66ee4d3e0; no voters:
I20260621 05:23:35.833714 17069 leader_election.cc:288] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260621 05:23:35.833766 17073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 3 FOLLOWER]: Leader election won for term 3
I20260621 05:23:35.833886 17073 raft_consensus.cc:686] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [term 3 LEADER]: Becoming Leader. State: Replica: e01df39404d24d239c5ac2c66ee4d3e0, State: Running, Role: LEADER
I20260621 05:23:35.833976 17073 consensus_queue.cc:237] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 32835 } }
I20260621 05:23:35.833976 17069 sys_catalog.cc:564] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:23:35.834270 17075 sys_catalog.cc:455] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 32835 } } }
I20260621 05:23:35.834359 17075 sys_catalog.cc:458] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [sys.catalog]: This master's current role is: LEADER
I20260621 05:23:35.834491 17074 sys_catalog.cc:455] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e01df39404d24d239c5ac2c66ee4d3e0. Latest consensus state: current_term: 3 leader_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01df39404d24d239c5ac2c66ee4d3e0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 32835 } } }
I20260621 05:23:35.834544 17074 sys_catalog.cc:458] T 00000000000000000000000000000000 P e01df39404d24d239c5ac2c66ee4d3e0 [sys.catalog]: This master's current role is: LEADER
I20260621 05:23:35.834676 17077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:23:35.836210 17077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:23:35.836586 17077 catalog_manager.cc:1259] Loaded cluster ID: bf8cba02be7a4f5d991cb74e7384edd3
I20260621 05:23:35.837143 17077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260621 05:23:35.837469 17077 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fef3e7ad9ad google::LogMessage::Fail() at ??:0
@ 0x7fef3e7b088a google::LogMessage::SendToLog() at ??:0
@ 0x7fef3e7ad4de google::LogMessage::Flush() at ??:0
@ 0x7fef3e7aea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fef466ce0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fef466d78b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fef4067b0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fef40670954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fef414d46db start_thread at ??:0
@ 0x7fef3d2ab61f clone at ??:0
W20260621 05:23:35.934437 17071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:35.934680 17071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:35.934725 17071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:35.936507 17071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:35.936610 17071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:35.936652 17071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:23:35.939528 17071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:43139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=34149
--tserver_master_addrs=127.2.32.62:32835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:35.939850 17071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:35.940106 17071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:35.945534 17105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:35.945693 17106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:35.945544 17108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:35.946799 17071 server_base.cc:1034] running on GCE node
I20260621 05:23:35.947100 17071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:35.947640 17071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:35.948853 17071 hybrid_clock.cc:648] HybridClock initialized: now 1782019415948854 us; error 47 us; skew 500 ppm
Jun 21 05:23:35 dist-test-slave-b357 krb5kdc[15930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019415, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:35.951913 17071 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:23:35.952507 17071 webserver.cc:460] Webserver started at http://127.2.32.1:34149/ using document root <none> and password file <none>
I20260621 05:23:35.952739 17071 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:35.952811 17071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:35.954205 17071 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:23:35.954973 17115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:35.955397 17071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:35.955606 17071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "481ca11e1b404475b9d31696996a6a33"
format_stamp: "Formatted at 2026-06-21 05:23:34 on dist-test-slave-b357"
server_key: "bdcd702c76863608cd79b2928a66363c"
server_key_iv: "c1037bba371917cf42f6f25b3246c07f"
server_key_version: "encryptionkey@0"
I20260621 05:23:35.956059 17071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:35.983382 17071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:35.985711 17071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:35.985822 17071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:35.986053 17071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:23:35.986515 17122 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 05:23:35.988529 17071 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 05:23:35.988611 17071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260621 05:23:35.988656 17071 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 05:23:35.989428 17071 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 05:23:35.989493 17071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260621 05:23:35.989545 17122 tablet_bootstrap.cc:492] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33: Bootstrap starting.
I20260621 05:23:35.999379 17071 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:43139
I20260621 05:23:35.999660 17229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:43139 every 8 connection(s)
I20260621 05:23:36.000110 17071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:23:36.005748 17122 log.cc:824] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:36.008893 17122 tablet_bootstrap.cc:492] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:23:36.009217 17122 tablet_bootstrap.cc:492] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33: Bootstrap complete.
I20260621 05:23:36.010074 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 17071
I20260621 05:23:36.010350 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:42189
--local_ip_for_outbound_sockets=127.2.32.2
--tserver_master_addrs=127.2.32.62:32835
--webserver_port=40425
--webserver_interface=127.2.32.2
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:23:36.010448 17122 ts_tablet_manager.cc:1397] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.011s
I20260621 05:23:36.011915 17122 raft_consensus.cc:348] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:36.012246 17122 raft_consensus.cc:729] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481ca11e1b404475b9d31696996a6a33, State: Initialized, Role: FOLLOWER
I20260621 05:23:36.012398 17122 consensus_queue.cc:260] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:36.012688 17122 ts_tablet_manager.cc:1428] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260621 05:23:36.109781 17232 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:23:36.001131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.1:39479)
0621 05:23:36.001456 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:36.001502 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:36.002084 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:36.109563 (+107479us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":131,"threads_started":1}
W20260621 05:23:36.110088 17230 heartbeater.cc:644] Failed to heartbeat to 127.2.32.62:32835 (0 consecutive failures): Network error: Failed to ping master at 127.2.32.62:32835: Client connection negotiation failed: client connection to 127.2.32.62:32835: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 05:23:36.110477 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:36.110357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.1:47499)
0621 05:23:36.110413 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:36.110450 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260621 05:23:36.110802 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:36.110674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.1:42949)
0621 05:23:36.110735 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:36.110774 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260621 05:23:36.110947 17230 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 05:23:36.113791 17234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:36.114022 17234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:36.114073 17234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:36.115334 17234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:36.115389 17234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:36.115432 17234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:23:36.117952 17234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:42189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=40425
--tserver_master_addrs=127.2.32.62:32835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:36.118400 17234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:36.118681 17234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:36.126734 17240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:36.126690 17241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:36.126690 17243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:36.127393 17234 server_base.cc:1034] running on GCE node
I20260621 05:23:36.127593 17234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:36.127813 17234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:36.128996 17234 hybrid_clock.cc:648] HybridClock initialized: now 1782019416128977 us; error 53 us; skew 500 ppm
Jun 21 05:23:36 dist-test-slave-b357 krb5kdc[15930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019416, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:36.132378 17234 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:23:36.132841 17234 webserver.cc:460] Webserver started at http://127.2.32.2:40425/ using document root <none> and password file <none>
I20260621 05:23:36.133071 17234 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:36.133193 17234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:36.134670 17234 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:23:36.135306 17250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:36.135762 17234 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:36.135838 17234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "956c8c30e25b4adf806e5cd32b30d39e"
format_stamp: "Formatted at 2026-06-21 05:23:34 on dist-test-slave-b357"
server_key: "2e6e1cbb9d0785b396ba899838bed749"
server_key_iv: "4e8d576fe664a6c282cd5934e8388af9"
server_key_version: "encryptionkey@0"
I20260621 05:23:36.136232 17234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:36.154125 17234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:36.157557 17234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:36.157903 17234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:36.158710 17234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:23:36.159232 17234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:23:36.159293 17234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:36.159340 17234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:23:36.159384 17234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:36.168486 17234 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:42189
I20260621 05:23:36.168561 17363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:42189 every 8 connection(s)
I20260621 05:23:36.168921 17234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:23:36.170305 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:36.169601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.2:41457)
0621 05:23:36.169755 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:36.169876 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
W20260621 05:23:36.170694 17364 heartbeater.cc:644] Failed to heartbeat to 127.2.32.62:32835 (0 consecutive failures): Network error: Failed to ping master at 127.2.32.62:32835: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
I20260621 05:23:36.171037 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:36.170951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.2:47695)
0621 05:23:36.170987 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:36.171017 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260621 05:23:36.171614 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:36.171483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.2:43145)
0621 05:23:36.171552 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:36.171594 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260621 05:23:36.171772 17364 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 05:23:36.177572 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 17234
I20260621 05:23:36.177817 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:41691
--local_ip_for_outbound_sockets=127.2.32.3
--tserver_master_addrs=127.2.32.62:32835
--webserver_port=45583
--webserver_interface=127.2.32.3
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:23:36.261514 17367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:36.261700 17367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:36.261722 17367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:36.262895 17367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:36.262938 17367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:36.262964 17367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:23:36.264405 17367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:41691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=45583
--tserver_master_addrs=127.2.32.62:32835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:36.264613 17367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:36.264856 17367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:36.269873 17372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:36.269861 17373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:36.269891 17367 server_base.cc:1034] running on GCE node
W20260621 05:23:36.269863 17375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:36.270258 17367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:36.270501 17367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:36.271682 17367 hybrid_clock.cc:648] HybridClock initialized: now 1782019416271655 us; error 45 us; skew 500 ppm
Jun 21 05:23:36 dist-test-slave-b357 krb5kdc[15930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019416, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:36.275018 17367 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:23:36.275662 17367 webserver.cc:460] Webserver started at http://127.2.32.3:45583/ using document root <none> and password file <none>
I20260621 05:23:36.275888 17367 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:36.276052 17367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:36.277561 17367 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:23:36.278507 17382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:36.278972 17367 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:23:36.279177 17367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "63d927b55814423a9280aae6fdc9083e"
format_stamp: "Formatted at 2026-06-21 05:23:34 on dist-test-slave-b357"
server_key: "7a8ffabf4e855250c06dbfd6afc27e0d"
server_key_iv: "bccdb0bcf4c4defd0888e04f67b541e9"
server_key_version: "encryptionkey@0"
I20260621 05:23:36.279809 17367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:36.304864 17367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:36.307210 17367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:36.307435 17367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:36.307703 17367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:23:36.308055 17367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:23:36.308112 17367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:36.308151 17367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:23:36.308197 17367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:36.314517 17367 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:41691
I20260621 05:23:36.314592 17495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:41691 every 8 connection(s)
I20260621 05:23:36.314939 17367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:23:36.315987 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:36.315639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.3:46357)
0621 05:23:36.315792 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:36.315858 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
W20260621 05:23:36.316437 17496 heartbeater.cc:644] Failed to heartbeat to 127.2.32.62:32835 (0 consecutive failures): Network error: Failed to ping master at 127.2.32.62:32835: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
I20260621 05:23:36.316918 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:36.316784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.3:37811)
0621 05:23:36.316863 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:36.316897 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260621 05:23:36.317607 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:36.317407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.3:36471)
0621 05:23:36.317521 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:36.317567 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260621 05:23:36.317839 17496 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 05:23:36.323197 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 17367
I20260621 05:23:37.111605 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:37.111372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.1:52275)
0621 05:23:37.111495 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:37.111535 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260621 05:23:37.172350 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:37.172112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.2:54681)
0621 05:23:37.172264 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:37.172298 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":58,"threads_started":1}
I20260621 05:23:37.318532 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:37.318192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.3:38903)
0621 05:23:37.318398 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:37.318460 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":66,"threads_started":1}
I20260621 05:23:37.597450 17503 raft_consensus.cc:482] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:23:37.597608 17503 raft_consensus.cc:504] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:37.597976 17503 leader_election.cc:288] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63d927b55814423a9280aae6fdc9083e (127.2.32.3:41691), 956c8c30e25b4adf806e5cd32b30d39e (127.2.32.2:42189)
Jun 21 05:23:37 dist-test-slave-b357 krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019415, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.3@KRBTEST.COM
Jun 21 05:23:37 dist-test-slave-b357 krb5kdc[15930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019415, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.2@KRBTEST.COM
I20260621 05:23:37.610477 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:37.598280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:41691 (local address 127.2.32.1:53755)
0621 05:23:37.598560 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:37.598573 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:37.598654 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:37.599627 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:37.599628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:37.599995 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:37.600072 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:37.600076 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:37.601576 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:37.601577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:37.602618 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:37.602621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:37.603551 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:37.603552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:37.603604 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:37.604137 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:37.604145 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:37.605720 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:37.608200 (+ 2480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:37.608207 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:37.608210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:37.608541 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:37.609043 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:37.609044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:37.609045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:37.609226 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:37.609773 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:37.609783 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:37.610038 (+ 255us) client_negotiation.cc:769] Sending connection context
0621 05:23:37.610167 (+ 129us) client_negotiation.cc:240] Negotiation successful
0621 05:23:37.610353 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":61,"thread_start_us":198,"threads_started":1}
I20260621 05:23:37.610536 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:37.598394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:53755 (local address 127.2.32.3:41691)
0621 05:23:37.598590 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:37.598604 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:37.598644 (+ 40us) server_negotiation.cc:407] Connection header received
0621 05:23:37.599395 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:37.599399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:37.599456 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:37.599621 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:37.600328 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:37.601446 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:37.602722 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:37.603455 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:37.603559 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:37.605876 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:37.605880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:37.605880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:37.605892 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:37.608098 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:37.608676 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:37.608681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:37.608681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:37.608856 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:37.609319 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:37.609320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:37.609321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:37.609652 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:37.609721 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:37.610251 (+ 530us) server_negotiation.cc:299] Negotiation successful
0621 05:23:37.610409 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":62,"threads_started":1}
I20260621 05:23:37.610677 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:37.598284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:51911 (local address 127.2.32.2:42189)
0621 05:23:37.598459 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:37.598471 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:37.598508 (+ 37us) server_negotiation.cc:407] Connection header received
0621 05:23:37.599057 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:37.599060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:37.599106 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:37.599286 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:37.600247 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:37.601266 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:37.602744 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:37.603260 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:37.603387 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:37.606351 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:37.606354 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:37.606354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:37.606363 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:37.608345 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:37.609031 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:37.609034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:37.609034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:37.609171 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:37.609525 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:37.609526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:37.609526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:37.610039 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:37.610128 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:37.610404 (+ 276us) server_negotiation.cc:299] Negotiation successful
0621 05:23:37.610547 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":66,"threads_started":1}
I20260621 05:23:37.611487 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:37.598174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:42189 (local address 127.2.32.1:51911)
0621 05:23:37.598261 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:37.598278 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:37.598358 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:37.599364 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:37.599367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:37.599857 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:37.600066 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:37.600073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:37.601507 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:37.601509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:37.602635 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:37.602639 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:37.603408 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:37.603409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:37.603545 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:37.604304 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:37.604310 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:37.606214 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:37.608442 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:37.608443 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:37.608443 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:37.608823 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:37.609318 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:37.609319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:37.609320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:37.609402 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:37.610175 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:37.610177 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:37.610238 (+ 61us) client_negotiation.cc:769] Sending connection context
0621 05:23:37.610318 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 05:23:37.610418 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":3}
W20260621 05:23:37.611904 17118 leader_election.cc:341] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63d927b55814423a9280aae6fdc9083e (127.2.32.3:41691): Not found: Tablet not found: 1f911dfb1db946ff9c222375339c883c
W20260621 05:23:37.612317 17116 leader_election.cc:341] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 956c8c30e25b4adf806e5cd32b30d39e (127.2.32.2:42189): Not found: Tablet not found: 1f911dfb1db946ff9c222375339c883c
I20260621 05:23:37.612407 17116 leader_election.cc:302] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 481ca11e1b404475b9d31696996a6a33; no voters: 63d927b55814423a9280aae6fdc9083e, 956c8c30e25b4adf806e5cd32b30d39e
I20260621 05:23:37.612552 17503 raft_consensus.cc:2726] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:23:38.112340 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:38.112102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.1:52405)
0621 05:23:38.112252 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:38.112291 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":56,"threads_started":1}
I20260621 05:23:38.173106 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:38.172878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.2:40203)
0621 05:23:38.173033 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:38.173070 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260621 05:23:38.319449 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:38.319212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.3:57509)
0621 05:23:38.319378 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:38.319416 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":88,"threads_started":1}
I20260621 05:23:39.113360 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:39.113043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.1:53331)
0621 05:23:39.113256 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:39.113308 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":109,"threads_started":1}
I20260621 05:23:39.173846 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:39.173605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.2:56853)
0621 05:23:39.173737 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:39.173804 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260621 05:23:39.320407 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:39.320102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.3:33899)
0621 05:23:39.320324 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:39.320367 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":92,"threads_started":1}
I20260621 05:23:39.759883 17513 raft_consensus.cc:482] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:23:39.760008 17513 raft_consensus.cc:504] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:39.760285 17513 leader_election.cc:288] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63d927b55814423a9280aae6fdc9083e (127.2.32.3:41691), 956c8c30e25b4adf806e5cd32b30d39e (127.2.32.2:42189)
W20260621 05:23:39.760712 17116 leader_election.cc:341] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 956c8c30e25b4adf806e5cd32b30d39e (127.2.32.2:42189): Not found: Tablet not found: 1f911dfb1db946ff9c222375339c883c
W20260621 05:23:39.760812 17118 leader_election.cc:341] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63d927b55814423a9280aae6fdc9083e (127.2.32.3:41691): Not found: Tablet not found: 1f911dfb1db946ff9c222375339c883c
I20260621 05:23:39.760865 17116 leader_election.cc:302] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 481ca11e1b404475b9d31696996a6a33; no voters: 63d927b55814423a9280aae6fdc9083e, 956c8c30e25b4adf806e5cd32b30d39e
I20260621 05:23:39.760962 17513 raft_consensus.cc:2726] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:23:40.114341 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:40.113995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.1:53179)
0621 05:23:40.114249 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:40.114291 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":120,"threads_started":1}
I20260621 05:23:40.174629 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:40.174367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.2:58993)
0621 05:23:40.174547 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:40.174580 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
I20260621 05:23:40.321398 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:40.321062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.3:50793)
0621 05:23:40.321280 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:40.321332 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":112,"threads_started":1}
I20260621 05:23:41.115590 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:41.115339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.1:58903)
0621 05:23:41.115497 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:41.115541 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":68,"threads_started":1}
I20260621 05:23:41.175709 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:41.175351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.2:40495)
0621 05:23:41.175554 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:41.175633 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":66,"threads_started":1}
I20260621 05:23:41.322383 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:41.322069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.3:44433)
0621 05:23:41.322277 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:41.322324 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":84,"threads_started":1}
I20260621 05:23:42.116632 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:42.116270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.1:40543)
0621 05:23:42.116492 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:42.116565 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":68,"threads_started":1}
I20260621 05:23:42.176862 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:42.176608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.2:48475)
0621 05:23:42.176773 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:42.176816 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":79,"threads_started":1}
I20260621 05:23:42.323477 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:42.323179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.3:35727)
0621 05:23:42.323350 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:42.323426 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
I20260621 05:23:42.951635 17523 raft_consensus.cc:482] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:23:42.951725 17523 raft_consensus.cc:504] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:42.951897 17523 leader_election.cc:288] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63d927b55814423a9280aae6fdc9083e (127.2.32.3:41691), 956c8c30e25b4adf806e5cd32b30d39e (127.2.32.2:42189)
W20260621 05:23:42.952286 17116 leader_election.cc:341] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 956c8c30e25b4adf806e5cd32b30d39e (127.2.32.2:42189): Not found: Tablet not found: 1f911dfb1db946ff9c222375339c883c
W20260621 05:23:42.952364 17118 leader_election.cc:341] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63d927b55814423a9280aae6fdc9083e (127.2.32.3:41691): Not found: Tablet not found: 1f911dfb1db946ff9c222375339c883c
I20260621 05:23:42.952409 17118 leader_election.cc:302] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 481ca11e1b404475b9d31696996a6a33; no voters: 63d927b55814423a9280aae6fdc9083e, 956c8c30e25b4adf806e5cd32b30d39e
I20260621 05:23:42.952493 17523 raft_consensus.cc:2726] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:23:43.117616 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:43.117270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.1:58277)
0621 05:23:43.117485 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:43.117550 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":115,"threads_started":1}
I20260621 05:23:43.177819 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:43.177512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.2:54153)
0621 05:23:43.177706 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:43.177757 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":86,"threads_started":1}
I20260621 05:23:43.324465 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:43.324091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.3:45687)
0621 05:23:43.324328 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:43.324395 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":83,"threads_started":1}
I20260621 05:23:44.118647 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:44.118391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.1:47485)
0621 05:23:44.118554 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:44.118597 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":77,"threads_started":1}
I20260621 05:23:44.178853 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:44.178457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.2:45113)
0621 05:23:44.178667 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:44.178732 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":63,"threads_started":1}
I20260621 05:23:44.325426 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:44.325100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.3:45071)
0621 05:23:44.325286 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:44.325365 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
I20260621 05:23:45.119704 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:45.119385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.1:42351)
0621 05:23:45.119570 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:45.119639 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":70,"threads_started":1}
I20260621 05:23:45.179957 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:45.179651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.2:51145)
0621 05:23:45.179845 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:45.179895 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":93,"threads_started":1}
I20260621 05:23:45.326462 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:45.326131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.3:43953)
0621 05:23:45.326347 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:45.326402 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":83,"threads_started":1}
I20260621 05:23:46.120636 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:46.120405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.1:58263)
0621 05:23:46.120548 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:46.120589 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260621 05:23:46.181023 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:46.180675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.2:34713)
0621 05:23:46.180907 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:46.180959 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":119,"threads_started":1}
I20260621 05:23:46.327512 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:46.327191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.3:52485)
0621 05:23:46.327394 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:46.327437 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":90,"threads_started":1}
I20260621 05:23:47.121706 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:47.121367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.1:60359)
0621 05:23:47.121608 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:47.121657 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":132,"threads_started":1}
I20260621 05:23:47.182053 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:47.181723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.2:45991)
0621 05:23:47.181951 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:47.181994 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":141,"threads_started":1}
I20260621 05:23:47.328411 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:47.328176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.3:41497)
0621 05:23:47.328326 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:47.328367 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260621 05:23:47.875192 17539 raft_consensus.cc:482] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:23:47.875313 17539 raft_consensus.cc:504] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d927b55814423a9280aae6fdc9083e" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41691 } } peers { permanent_uuid: "956c8c30e25b4adf806e5cd32b30d39e" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 42189 } } peers { permanent_uuid: "481ca11e1b404475b9d31696996a6a33" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 43139 } }
I20260621 05:23:47.875468 17539 leader_election.cc:288] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63d927b55814423a9280aae6fdc9083e (127.2.32.3:41691), 956c8c30e25b4adf806e5cd32b30d39e (127.2.32.2:42189)
W20260621 05:23:47.875860 17118 leader_election.cc:341] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63d927b55814423a9280aae6fdc9083e (127.2.32.3:41691): Not found: Tablet not found: 1f911dfb1db946ff9c222375339c883c
W20260621 05:23:47.875975 17116 leader_election.cc:341] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 956c8c30e25b4adf806e5cd32b30d39e (127.2.32.2:42189): Not found: Tablet not found: 1f911dfb1db946ff9c222375339c883c
I20260621 05:23:47.876034 17116 leader_election.cc:302] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 481ca11e1b404475b9d31696996a6a33; no voters: 63d927b55814423a9280aae6fdc9083e, 956c8c30e25b4adf806e5cd32b30d39e
I20260621 05:23:47.876145 17539 raft_consensus.cc:2726] T 1f911dfb1db946ff9c222375339c883c P 481ca11e1b404475b9d31696996a6a33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:23:48.122767 17540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:48.122419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.1:42283)
0621 05:23:48.122653 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:48.122700 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":115,"threads_started":1}
I20260621 05:23:48.183054 17541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:48.182695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.2:39311)
0621 05:23:48.182879 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:48.182992 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
I20260621 05:23:48.329320 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:48.329077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.3:55015)
0621 05:23:48.329230 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:48.329274 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260621 05:23:49.123891 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:49.123556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.1:46861)
0621 05:23:49.123748 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:49.123841 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":97,"threads_started":1}
I20260621 05:23:49.184093 17544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:49.183723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.2:60389)
0621 05:23:49.183976 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:49.184030 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":139,"threads_started":1}
I20260621 05:23:49.330319 17545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:49.330047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.3:43049)
0621 05:23:49.330233 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:49.330278 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":91,"threads_started":1}
I20260621 05:23:50.124953 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:50.124617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.1:56823)
0621 05:23:50.124823 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:50.124885 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":87,"threads_started":1}
I20260621 05:23:50.185053 17547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:50.184733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.2:40447)
0621 05:23:50.184929 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:50.184992 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":62,"threads_started":1}
I20260621 05:23:50.331094 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:50.330847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.3:44935)
0621 05:23:50.331010 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:50.331049 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":79,"threads_started":1}
I20260621 05:23:51.125991 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:51.125737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.1:35229)
0621 05:23:51.125904 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:51.125937 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
I20260621 05:23:51.186043 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:51.185711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.2:54127)
0621 05:23:51.185916 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:51.185982 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260621 05:23:51.328648 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 17071
I20260621 05:23:51.331879 17551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:51.331670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:32835 (local address 127.2.32.3:33395)
0621 05:23:51.331807 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:51.331838 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260621 05:23:51.333675 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 17234
I20260621 05:23:51.338635 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 17367
2026-06-21T05:23:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18834 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:23:51 dist-test-slave-b357 krb5kdc[17554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:23:51 dist-test-slave-b357 krb5kdc[17554](info): set up 2 sockets
Jun 21 05:23:51 dist-test-slave-b357 krb5kdc[17554](info): commencing operation
krb5kdc: starting...
W20260621 05:23:52.395783 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:23:52 dist-test-slave-b357 krb5kdc[17554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:23:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:23:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:23:52.520169 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:40701
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:40701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:23:52.599956 17567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:52.600152 17567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:52.600173 17567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:52.601290 17567 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 05:23:52.601328 17567 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 05:23:52.601341 17567 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 05:23:52.601352 17567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:23:52.601365 17567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:52.601377 17567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:52.601389 17567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:23:52.601401 17567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:23:52.602869 17567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:40701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:40701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:52.603104 17567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:52.603401 17567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:52.609760 17572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:52.609793 17567 server_base.cc:1034] running on GCE node
W20260621 05:23:52.609841 17573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:52.609843 17575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:52.610317 17567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:52.610561 17567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:52.611717 17567 hybrid_clock.cc:648] HybridClock initialized: now 1782019432611706 us; error 34 us; skew 500 ppm
Jun 21 05:23:52 dist-test-slave-b357 krb5kdc[17554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019432, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:52.614461 17567 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:23:52.615020 17567 webserver.cc:460] Webserver started at http://127.2.32.62:43529/ using document root <none> and password file <none>
I20260621 05:23:52.615337 17567 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:52.615432 17567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:52.615576 17567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:23:52.616674 17567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "cb4f712f75ca4b649cf74281f34999e1"
format_stamp: "Formatted at 2026-06-21 05:23:52 on dist-test-slave-b357"
server_key: "ae9ec513fcf2b6d2a85e412b02e71fa9"
server_key_iv: "2a08e8ce492227228a9f81f632e2df00"
server_key_version: "encryptionkey@0"
I20260621 05:23:52.617074 17567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "cb4f712f75ca4b649cf74281f34999e1"
format_stamp: "Formatted at 2026-06-21 05:23:52 on dist-test-slave-b357"
server_key: "ae9ec513fcf2b6d2a85e412b02e71fa9"
server_key_iv: "2a08e8ce492227228a9f81f632e2df00"
server_key_version: "encryptionkey@0"
I20260621 05:23:52.618685 17567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:23:52.619668 17582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:52.619925 17567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:23:52.620016 17567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "cb4f712f75ca4b649cf74281f34999e1"
format_stamp: "Formatted at 2026-06-21 05:23:52 on dist-test-slave-b357"
server_key: "ae9ec513fcf2b6d2a85e412b02e71fa9"
server_key_iv: "2a08e8ce492227228a9f81f632e2df00"
server_key_version: "encryptionkey@0"
I20260621 05:23:52.620067 17567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:52.647078 17567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:52.649408 17567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:52.649519 17567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:52.654089 17567 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:40701
I20260621 05:23:52.654354 17634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:40701 every 8 connection(s)
I20260621 05:23:52.654623 17567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:23:52.655450 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 17567
I20260621 05:23:52.655555 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:23:52.655795 17635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:23:52.655836 2176 external_mini_cluster.cc:1428] Setting key 84b4ef39d6d89cf882746b0128cd3583
I20260621 05:23:52.659492 17635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1: Bootstrap starting.
I20260621 05:23:52.660192 17635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1: Neither blocks nor log segments found. Creating new log.
I20260621 05:23:52.660497 17635 log.cc:824] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:52.661245 17635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1: No bootstrap required, opened a new log
I20260621 05:23:52.662979 17635 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f712f75ca4b649cf74281f34999e1" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40701 } }
I20260621 05:23:52.663133 17635 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:23:52.663213 17635 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb4f712f75ca4b649cf74281f34999e1, State: Initialized, Role: FOLLOWER
Jun 21 05:23:52 dist-test-slave-b357 krb5kdc[17554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:52.663347 17635 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f712f75ca4b649cf74281f34999e1" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40701 } }
I20260621 05:23:52.663424 17635 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:23:52.663445 17635 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:23:52.663509 17635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:23:52.664188 17635 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f712f75ca4b649cf74281f34999e1" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40701 } }
I20260621 05:23:52.664321 17635 leader_election.cc:302] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb4f712f75ca4b649cf74281f34999e1; no voters:
I20260621 05:23:52.664485 17635 leader_election.cc:288] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:23:52.664548 17640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:23:52.664737 17640 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 1 LEADER]: Becoming Leader. State: Replica: cb4f712f75ca4b649cf74281f34999e1, State: Running, Role: LEADER
I20260621 05:23:52.664791 17635 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:23:52.664964 17640 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f712f75ca4b649cf74281f34999e1" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40701 } }
I20260621 05:23:52.665504 17641 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb4f712f75ca4b649cf74281f34999e1. Latest consensus state: current_term: 1 leader_uuid: "cb4f712f75ca4b649cf74281f34999e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f712f75ca4b649cf74281f34999e1" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40701 } } }
I20260621 05:23:52.665603 17641 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [sys.catalog]: This master's current role is: LEADER
I20260621 05:23:52.666116 17649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:23:52.666707 17649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:23:52.667104 17642 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb4f712f75ca4b649cf74281f34999e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f712f75ca4b649cf74281f34999e1" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40701 } } }
I20260621 05:23:52.667416 17642 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [sys.catalog]: This master's current role is: LEADER
I20260621 05:23:52.667624 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:52.656477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58314 (local address 127.2.32.62:40701)
0621 05:23:52.656711 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:52.656721 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:52.656750 (+ 29us) server_negotiation.cc:407] Connection header received
0621 05:23:52.657313 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:52.657320 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:52.657401 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:52.657569 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:52.658541 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:52.659568 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:52.660672 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:52.661227 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:52.661353 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:52.663812 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:52.663816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:52.663816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:52.663827 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:52.665744 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:52.666171 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:52.666175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:52.666175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:52.666281 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:52.666499 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:52.666500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:52.666500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:52.666724 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:52.666904 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:52.667296 (+ 392us) server_negotiation.cc:299] Negotiation successful
0621 05:23:52.667430 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":104,"threads_started":1}
I20260621 05:23:52.668426 17649 catalog_manager.cc:1347] Generated new cluster ID: 0c656a092a7044f6b9b98ad26355ee4a
I20260621 05:23:52.668483 17649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:23:52.682000 17649 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:23:52.682529 17649 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:23:52.686648 17649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1: Generated new TSK 0
I20260621 05:23:52.686820 17649 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
I20260621 05:23:52.718550 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:40701
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:23:52.803887 17661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:52.804104 17661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:52.804141 17661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:52.805223 17661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:52.805284 17661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:52.805330 17661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:23:52.806741 17661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:52.806988 17661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:52.807258 17661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:52.812095 17669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:52.812160 17667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:52.812160 17666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:52.812222 17661 server_base.cc:1034] running on GCE node
I20260621 05:23:52.812536 17661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:52.812770 17661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:52.814009 17661 hybrid_clock.cc:648] HybridClock initialized: now 1782019432813974 us; error 41 us; skew 500 ppm
Jun 21 05:23:52 dist-test-slave-b357 krb5kdc[17554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019432, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:52.817512 17661 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:23:52.817931 17661 webserver.cc:460] Webserver started at http://127.2.32.1:38945/ using document root <none> and password file <none>
I20260621 05:23:52.818164 17661 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:52.818235 17661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:52.818387 17661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:23:52.819437 17661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "11ba9033670a40f3b1dcd0cb865fa607"
format_stamp: "Formatted at 2026-06-21 05:23:52 on dist-test-slave-b357"
server_key: "01c6bdf916ebb244e1e9bb0f1d90f54f"
server_key_iv: "f5380ecc90cbc280d3b1e531e7ddf343"
server_key_version: "encryptionkey@0"
I20260621 05:23:52.819772 17661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "11ba9033670a40f3b1dcd0cb865fa607"
format_stamp: "Formatted at 2026-06-21 05:23:52 on dist-test-slave-b357"
server_key: "01c6bdf916ebb244e1e9bb0f1d90f54f"
server_key_iv: "f5380ecc90cbc280d3b1e531e7ddf343"
server_key_version: "encryptionkey@0"
I20260621 05:23:52.821326 17661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:23:52.822184 17676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:52.822494 17661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:52.822593 17661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "11ba9033670a40f3b1dcd0cb865fa607"
format_stamp: "Formatted at 2026-06-21 05:23:52 on dist-test-slave-b357"
server_key: "01c6bdf916ebb244e1e9bb0f1d90f54f"
server_key_iv: "f5380ecc90cbc280d3b1e531e7ddf343"
server_key_version: "encryptionkey@0"
I20260621 05:23:52.822680 17661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:52.833675 17661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:52.835816 17661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:52.835975 17661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:52.836195 17661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:23:52.836485 17661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:23:52.836539 17661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:52.836575 17661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:23:52.836608 17661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:52.843122 17661 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:36527
I20260621 05:23:52.843215 17789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:36527 every 8 connection(s)
I20260621 05:23:52.843508 17661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:23:52 dist-test-slave-b357 krb5kdc[17554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019432, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:52.853555 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 17661
I20260621 05:23:52.853699 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:23:52.853909 2176 external_mini_cluster.cc:1428] Setting key 2bec97d33cc1986ecbc3912537badf65
I20260621 05:23:52.854156 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:52.844529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:37183 (local address 127.2.32.62:40701)
0621 05:23:52.844626 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:52.844629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:52.844791 (+ 162us) server_negotiation.cc:407] Connection header received
0621 05:23:52.845206 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:52.845207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:52.845219 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:52.845279 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:52.846159 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:52.846830 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:52.848402 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:52.848638 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:52.848707 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:52.850927 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:52.850928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:52.850929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:52.850936 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:52.852719 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:52.853146 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:52.853147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:52.853148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:52.853206 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:52.853470 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:52.853471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:52.853471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:52.853596 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:52.853667 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:52.853970 (+ 303us) server_negotiation.cc:299] Negotiation successful
0621 05:23:52.854065 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 05:23:52.854334 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:52.844442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.1:37183)
0621 05:23:52.844625 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:52.844665 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:52.845108 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:52.845330 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:52.845333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:52.845852 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:52.846025 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:52.846030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:52.846989 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:52.846990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:52.848156 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:52.848160 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:52.848750 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:52.848751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:52.848889 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:52.849467 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:52.849475 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:52.850826 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:52.852833 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:52.852839 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:52.852841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:52.853053 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:52.853274 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:52.853275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:52.853276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:52.853398 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:52.853670 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:52.853678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:52.853832 (+ 154us) client_negotiation.cc:769] Sending connection context
0621 05:23:52.853949 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 05:23:52.854110 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":99,"threads_started":1}
I20260621 05:23:52.854903 17790 heartbeater.cc:347] Connected to a master server at 127.2.32.62:40701
I20260621 05:23:52.855024 17790 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:52.855239 17790 heartbeater.cc:511] Master 127.2.32.62:40701 requested a full tablet report, sending...
I20260621 05:23:52.855821 17599 ts_manager.cc:194] Registered new tserver with Master: 11ba9033670a40f3b1dcd0cb865fa607 (127.2.32.1:36527)
I20260621 05:23:52.856591 17599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:37183
WARNING: no policy specified for kudu/127.2.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
I20260621 05:23:52.882218 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:40701
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:23:52.966202 17795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:52.966470 17795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:52.966499 17795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:52.967754 17795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:52.967798 17795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:52.967826 17795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:23:52.969249 17795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:52.969477 17795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:52.969720 17795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:52.974974 17803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:52.975003 17800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:52.974992 17801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:52.975311 17795 server_base.cc:1034] running on GCE node
I20260621 05:23:52.975559 17795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:52.975790 17795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:52.976955 17795 hybrid_clock.cc:648] HybridClock initialized: now 1782019432976945 us; error 30 us; skew 500 ppm
Jun 21 05:23:52 dist-test-slave-b357 krb5kdc[17554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019432, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:52.979897 17795 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:23:52.980401 17795 webserver.cc:460] Webserver started at http://127.2.32.2:33649/ using document root <none> and password file <none>
I20260621 05:23:52.980670 17795 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:52.980726 17795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:52.980870 17795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:23:52.982105 17795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "be235d3c549d44f0a9b4924326f225cb"
format_stamp: "Formatted at 2026-06-21 05:23:52 on dist-test-slave-b357"
server_key: "5483910a424db3844a921432c28d7377"
server_key_iv: "bafd5d9517524dc182486b91aa4831ee"
server_key_version: "encryptionkey@0"
I20260621 05:23:52.982512 17795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "be235d3c549d44f0a9b4924326f225cb"
format_stamp: "Formatted at 2026-06-21 05:23:52 on dist-test-slave-b357"
server_key: "5483910a424db3844a921432c28d7377"
server_key_iv: "bafd5d9517524dc182486b91aa4831ee"
server_key_version: "encryptionkey@0"
I20260621 05:23:52.984110 17795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:23:52.985226 17810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:52.985646 17795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:23:52.985791 17795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "be235d3c549d44f0a9b4924326f225cb"
format_stamp: "Formatted at 2026-06-21 05:23:52 on dist-test-slave-b357"
server_key: "5483910a424db3844a921432c28d7377"
server_key_iv: "bafd5d9517524dc182486b91aa4831ee"
server_key_version: "encryptionkey@0"
I20260621 05:23:52.985904 17795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:53.008796 17795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:53.011184 17795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:53.011418 17795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:53.011711 17795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:23:53.012102 17795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:23:53.012162 17795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:53.012207 17795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:23:53.012236 17795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:53.020138 17795 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:37849
I20260621 05:23:53.020385 17923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:37849 every 8 connection(s)
I20260621 05:23:53.020632 17795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:23:53.027496 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 17795
I20260621 05:23:53.027675 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:23:53.027945 2176 external_mini_cluster.cc:1428] Setting key 7ea9bb20686799ae60b83e18e8a7595d
Jun 21 05:23:53 dist-test-slave-b357 krb5kdc[17554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019432, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:53.032743 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:53.021630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:46599 (local address 127.2.32.62:40701)
0621 05:23:53.021699 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:53.021701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:53.021845 (+ 144us) server_negotiation.cc:407] Connection header received
0621 05:23:53.022487 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:53.022488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:53.022500 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:53.022581 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:53.023766 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.024332 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.025605 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.025838 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.025914 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:53.028837 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:53.028839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:53.028840 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:53.028849 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:53.030766 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:53.031321 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:53.031322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:53.031323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:53.031403 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:53.031792 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:53.031794 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:53.031794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:53.031938 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:53.032016 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:53.032592 (+ 576us) server_negotiation.cc:299] Negotiation successful
0621 05:23:53.032675 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 05:23:53.033020 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:53.021496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.2:46599)
0621 05:23:53.021679 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:53.021726 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:53.022341 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:53.022656 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:53.022660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:53.023376 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:53.023657 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:53.023662 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.024443 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.024444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:53.025491 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:53.025494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.025963 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.025965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:53.026103 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:53.026800 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:53.026808 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:53.028646 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:53.030886 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:53.030893 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:53.030895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:53.031233 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:53.031489 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:53.031490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:53.031490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:53.031660 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:53.032083 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:53.032095 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:53.032419 (+ 324us) client_negotiation.cc:769] Sending connection context
0621 05:23:53.032537 (+ 118us) client_negotiation.cc:240] Negotiation successful
0621 05:23:53.032736 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
I20260621 05:23:53.033488 17924 heartbeater.cc:347] Connected to a master server at 127.2.32.62:40701
I20260621 05:23:53.033605 17924 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:53.033804 17924 heartbeater.cc:511] Master 127.2.32.62:40701 requested a full tablet report, sending...
I20260621 05:23:53.034215 17599 ts_manager.cc:194] Registered new tserver with Master: be235d3c549d44f0a9b4924326f225cb (127.2.32.2:37849)
I20260621 05:23:53.034600 17599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:46599
WARNING: no policy specified for kudu/127.2.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
I20260621 05:23:53.055907 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:40701
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:23:53.136909 17929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:23:53.137097 17929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:23:53.137120 17929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:23:53.138244 17929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:23:53.138285 17929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:23:53.138312 17929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:23:53.139752 17929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:23:53.139982 17929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:23:53.140205 17929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:23:53.145543 17937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:53.145586 17934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:23:53.145532 17935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:23:53.145814 17929 server_base.cc:1034] running on GCE node
I20260621 05:23:53.146075 17929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:23:53.146381 17929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:23:53.147553 17929 hybrid_clock.cc:648] HybridClock initialized: now 1782019433147523 us; error 52 us; skew 500 ppm
Jun 21 05:23:53 dist-test-slave-b357 krb5kdc[17554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019433, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:23:53.150228 17929 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:23:53.150712 17929 webserver.cc:460] Webserver started at http://127.2.32.3:37085/ using document root <none> and password file <none>
I20260621 05:23:53.150933 17929 fs_manager.cc:359] Metadata directory not provided
I20260621 05:23:53.150998 17929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:23:53.151193 17929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:23:53.152187 17929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "d7bb06947ddb440fafc0bf38fa8b1037"
format_stamp: "Formatted at 2026-06-21 05:23:53 on dist-test-slave-b357"
server_key: "b4bce4edab6920e49a556c9516153204"
server_key_iv: "76cad499582e59b19331684fa617939a"
server_key_version: "encryptionkey@0"
I20260621 05:23:53.152563 17929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "d7bb06947ddb440fafc0bf38fa8b1037"
format_stamp: "Formatted at 2026-06-21 05:23:53 on dist-test-slave-b357"
server_key: "b4bce4edab6920e49a556c9516153204"
server_key_iv: "76cad499582e59b19331684fa617939a"
server_key_version: "encryptionkey@0"
I20260621 05:23:53.153921 17929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:23:53.154742 17944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:53.154968 17929 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:23:53.155072 17929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "d7bb06947ddb440fafc0bf38fa8b1037"
format_stamp: "Formatted at 2026-06-21 05:23:53 on dist-test-slave-b357"
server_key: "b4bce4edab6920e49a556c9516153204"
server_key_iv: "76cad499582e59b19331684fa617939a"
server_key_version: "encryptionkey@0"
I20260621 05:23:53.155181 17929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:23:53.174445 17929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:23:53.177229 17929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:23:53.177376 17929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:23:53.177631 17929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:23:53.178007 17929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:23:53.178062 17929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:53.178100 17929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:23:53.178128 17929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:23:53.186359 17929 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:40135
I20260621 05:23:53.186662 18057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:40135 every 8 connection(s)
I20260621 05:23:53.186800 17929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:23:53.191258 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 17929
I20260621 05:23:53.191381 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:23:53.191579 2176 external_mini_cluster.cc:1428] Setting key 9e96cec781430aceb07f46bf3c3f182e
Jun 21 05:23:53 dist-test-slave-b357 krb5kdc[17554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019433, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:23:53.198292 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:53.187777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.3:58279)
0621 05:23:53.187978 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:53.188024 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:53.188754 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:53.189115 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:53.189119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:53.189790 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:23:53.189997 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:53.190001 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.190727 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.190728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:53.191903 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:53.191906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.192348 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.192349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:53.192479 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:53.192997 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:23:53.193006 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:23:53.194753 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:23:53.196708 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:53.196715 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:53.196717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:53.197065 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:53.197267 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:23:53.197268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:23:53.197268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:23:53.197392 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:23:53.197666 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:23:53.197672 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:23:53.197819 (+ 147us) client_negotiation.cc:769] Sending connection context
0621 05:23:53.197936 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 05:23:53.198093 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":122,"threads_started":1}
I20260621 05:23:53.198444 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:53.187791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:58279 (local address 127.2.32.62:40701)
0621 05:23:53.187855 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:53.187857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:53.188106 (+ 249us) server_negotiation.cc:407] Connection header received
0621 05:23:53.188914 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:53.188916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:53.188929 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:53.189004 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:53.190087 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.190629 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.191993 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.192276 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.192339 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:53.194850 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:53.194851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:53.194852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:53.194861 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:53.196627 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:53.197137 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:53.197138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:53.197139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:53.197194 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:53.197470 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:53.197471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:53.197471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:53.197581 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:53.197652 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:53.198061 (+ 409us) server_negotiation.cc:299] Negotiation successful
0621 05:23:53.198238 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 05:23:53.199733 18058 heartbeater.cc:347] Connected to a master server at 127.2.32.62:40701
I20260621 05:23:53.199903 18058 heartbeater.cc:464] Registering TS with master...
I20260621 05:23:53.200188 18058 heartbeater.cc:511] Master 127.2.32.62:40701 requested a full tablet report, sending...
I20260621 05:23:53.200655 17599 ts_manager.cc:194] Registered new tserver with Master: d7bb06947ddb440fafc0bf38fa8b1037 (127.2.32.3:40135)
I20260621 05:23:53.201236 17599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:58279
I20260621 05:23:53.203680 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:23:53.213172 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:53.205304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58330 (local address 127.2.32.62:40701)
0621 05:23:53.205343 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:53.205344 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:53.205693 (+ 349us) server_negotiation.cc:407] Connection header received
0621 05:23:53.205795 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:53.205796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:53.205812 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:53.205879 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:23:53.206510 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.207422 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.208662 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.209038 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.209124 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:53.210520 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:23:53.210521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:23:53.210521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:23:53.210529 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:23:53.212002 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:53.212367 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:53.212368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:53.212368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:53.212419 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:23:53.212625 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:23:53.212625 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:23:53.212626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:23:53.212777 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:23:53.212883 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:53.212987 (+ 104us) server_negotiation.cc:299] Negotiation successful
0621 05:23:53.213069 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:23:53.214928 17599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:23:53.217723 17599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:23:53.225257 18074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:53.221158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33092 (local address 127.2.32.2:37849)
0621 05:23:53.221349 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:53.221355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:53.221555 (+ 200us) server_negotiation.cc:407] Connection header received
0621 05:23:53.221629 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:53.221631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:53.221646 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:53.221717 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:23:53.221998 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.222843 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.224214 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.224835 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.224911 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:53.225005 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:53.225128 (+ 123us) server_negotiation.cc:299] Negotiation successful
0621 05:23:53.225167 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20260621 05:23:53.225278 18070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:53.221023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:37849 (local address 127.0.0.1:33092)
0621 05:23:53.221436 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:53.221445 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:53.221497 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:53.221790 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:53.221790 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:53.221792 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:23:53.221822 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:53.221823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.222961 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.222962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:53.224114 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:53.224116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.224928 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.224929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:53.225007 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:53.225021 (+ 14us) client_negotiation.cc:769] Sending connection context
0621 05:23:53.225101 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 05:23:53.225200 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":93,"thread_start_us":78,"threads_started":1}
I20260621 05:23:53.225543 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:53.220808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:36527 (local address 127.0.0.1:44758)
0621 05:23:53.221272 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:53.221284 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:53.221346 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:53.221888 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:53.221889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:53.221891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:23:53.221925 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:53.221926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.223116 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.223117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:53.224429 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:53.224432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.225322 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.225323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:53.225388 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:53.225392 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:23:53.225440 (+ 48us) client_negotiation.cc:240] Negotiation successful
0621 05:23:53.225477 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":30,"thread_start_us":69,"threads_started":1}
I20260621 05:23:53.225780 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:53.220951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44758 (local address 127.2.32.1:36527)
0621 05:23:53.221725 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:53.221728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:53.221735 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:23:53.221790 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:53.221792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:53.221805 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:53.221886 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:23:53.221982 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.223027 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.224535 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.225181 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.225241 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:53.225289 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:53.225652 (+ 363us) server_negotiation.cc:299] Negotiation successful
0621 05:23:53.225707 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":117,"threads_started":1}
I20260621 05:23:53.226009 17858 tablet_service.cc:1467] Processing CreateTablet for tablet 1142ce01f9e84f61b8ec9e271cc1f98b (DEFAULT_TABLE table=test-table [id=afdd56bd700445a4a0ae385416c8edc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:23:53.226298 18069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:53.220808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:40135 (local address 127.0.0.1:60236)
0621 05:23:53.221166 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:53.221202 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:53.221266 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:53.221634 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:53.221636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:53.221639 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:23:53.221714 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:53.221716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.223044 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.223045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:53.224438 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:53.224440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.226024 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.226024 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:53.226117 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:53.226120 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:23:53.226177 (+ 57us) client_negotiation.cc:240] Negotiation successful
0621 05:23:53.226206 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":89,"thread_start_us":64,"threads_started":1}
I20260621 05:23:53.226341 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:53.221003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60236 (local address 127.2.32.3:40135)
0621 05:23:53.221429 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:53.221433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:53.221451 (+ 18us) server_negotiation.cc:407] Connection header received
0621 05:23:53.221519 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:53.221521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:53.221538 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:53.221616 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:23:53.221795 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.222951 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.224520 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.225220 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.225288 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:53.225995 (+ 707us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:53.226222 (+ 227us) server_negotiation.cc:299] Negotiation successful
0621 05:23:53.226267 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":79,"threads_started":1}
I20260621 05:23:53.226449 17858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1142ce01f9e84f61b8ec9e271cc1f98b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:23:53.226837 17992 tablet_service.cc:1467] Processing CreateTablet for tablet 1142ce01f9e84f61b8ec9e271cc1f98b (DEFAULT_TABLE table=test-table [id=afdd56bd700445a4a0ae385416c8edc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:23:53.227201 17992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1142ce01f9e84f61b8ec9e271cc1f98b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:23:53.229461 18075 tablet_bootstrap.cc:492] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb: Bootstrap starting.
I20260621 05:23:53.229717 18076 tablet_bootstrap.cc:492] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037: Bootstrap starting.
I20260621 05:23:53.229962 17724 tablet_service.cc:1467] Processing CreateTablet for tablet 1142ce01f9e84f61b8ec9e271cc1f98b (DEFAULT_TABLE table=test-table [id=afdd56bd700445a4a0ae385416c8edc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:23:53.230274 17724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1142ce01f9e84f61b8ec9e271cc1f98b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:23:53.230366 18076 tablet_bootstrap.cc:654] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037: Neither blocks nor log segments found. Creating new log.
I20260621 05:23:53.230896 18076 log.cc:824] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:53.231731 18075 tablet_bootstrap.cc:654] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb: Neither blocks nor log segments found. Creating new log.
I20260621 05:23:53.232185 18076 tablet_bootstrap.cc:492] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037: No bootstrap required, opened a new log
I20260621 05:23:53.232264 18076 ts_tablet_manager.cc:1397] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260621 05:23:53.232314 18075 log.cc:824] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:53.233094 18075 tablet_bootstrap.cc:492] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb: No bootstrap required, opened a new log
I20260621 05:23:53.233163 18075 ts_tablet_manager.cc:1397] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260621 05:23:53.233635 18079 tablet_bootstrap.cc:492] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607: Bootstrap starting.
I20260621 05:23:53.234087 18076 raft_consensus.cc:348] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } }
I20260621 05:23:53.234274 18076 raft_consensus.cc:374] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:23:53.234319 18076 raft_consensus.cc:729] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7bb06947ddb440fafc0bf38fa8b1037, State: Initialized, Role: FOLLOWER
I20260621 05:23:53.234459 18079 tablet_bootstrap.cc:654] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607: Neither blocks nor log segments found. Creating new log.
I20260621 05:23:53.234431 18076 consensus_queue.cc:260] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } }
I20260621 05:23:53.234723 18076 ts_tablet_manager.cc:1428] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:23:53.234844 18058 heartbeater.cc:503] Master 127.2.32.62:40701 was elected leader, sending a full tablet report...
I20260621 05:23:53.235297 18075 raft_consensus.cc:348] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } }
I20260621 05:23:53.235469 18075 raft_consensus.cc:374] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:23:53.235510 18075 raft_consensus.cc:729] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be235d3c549d44f0a9b4924326f225cb, State: Initialized, Role: FOLLOWER
I20260621 05:23:53.235626 18075 consensus_queue.cc:260] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } }
I20260621 05:23:53.235906 17924 heartbeater.cc:503] Master 127.2.32.62:40701 was elected leader, sending a full tablet report...
I20260621 05:23:53.236057 18079 log.cc:824] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607: Log is configured to *not* fsync() on all Append() calls
I20260621 05:23:53.235898 18075 ts_tablet_manager.cc:1428] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 05:23:53.237012 18079 tablet_bootstrap.cc:492] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607: No bootstrap required, opened a new log
I20260621 05:23:53.237098 18079 ts_tablet_manager.cc:1397] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260621 05:23:53.238360 18079 raft_consensus.cc:348] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } }
I20260621 05:23:53.238503 18079 raft_consensus.cc:374] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:23:53.238545 18079 raft_consensus.cc:729] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11ba9033670a40f3b1dcd0cb865fa607, State: Initialized, Role: FOLLOWER
I20260621 05:23:53.238683 18079 consensus_queue.cc:260] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } }
I20260621 05:23:53.238958 18079 ts_tablet_manager.cc:1428] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:23:53.238999 17790 heartbeater.cc:503] Master 127.2.32.62:40701 was elected leader, sending a full tablet report...
W20260621 05:23:53.271293 17925 tablet.cc:2369] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:23:53.287216 18083 raft_consensus.cc:482] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:23:53.287364 18083 raft_consensus.cc:504] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } }
I20260621 05:23:53.287689 18083 leader_election.cc:288] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7bb06947ddb440fafc0bf38fa8b1037 (127.2.32.3:40135), be235d3c549d44f0a9b4924326f225cb (127.2.32.2:37849)
I20260621 05:23:53.292047 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:53.288133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:40135 (local address 127.2.32.1:34967)
0621 05:23:53.288248 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:53.288266 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:53.288354 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:53.288571 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:53.288573 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:53.288574 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:23:53.288652 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:53.288654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.289623 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.289624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:53.290853 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:53.290856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.291710 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.291711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:53.291798 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:53.291800 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:23:53.291864 (+ 64us) client_negotiation.cc:240] Negotiation successful
0621 05:23:53.291937 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":16}
I20260621 05:23:53.292059 18084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:53.288372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:37849 (local address 127.2.32.1:49187)
0621 05:23:53.288568 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:53.288579 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:23:53.288639 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:23:53.288832 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:23:53.288833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:23:53.288835 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:23:53.288894 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:53.288895 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.289697 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.289697 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:53.290853 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:23:53.290856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.291710 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.291711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:23:53.291794 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:53.291799 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:23:53.291864 (+ 65us) client_negotiation.cc:240] Negotiation successful
0621 05:23:53.291940 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":27,"thread_start_us":130,"threads_started":1}
I20260621 05:23:53.292217 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:53.288268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:34967 (local address 127.2.32.3:40135)
0621 05:23:53.288324 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:53.288326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:53.288337 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:23:53.288457 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:53.288459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:53.288476 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:53.288547 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:23:53.288724 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.289497 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.290979 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.291580 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.291683 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:53.291695 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:53.291972 (+ 277us) server_negotiation.cc:299] Negotiation successful
0621 05:23:53.292019 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 05:23:53.292218 18074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:53.288452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:49187 (local address 127.2.32.2:37849)
0621 05:23:53.288497 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:53.288499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:53.288647 (+ 148us) server_negotiation.cc:407] Connection header received
0621 05:23:53.288723 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:53.288724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:53.288736 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:53.288798 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:23:53.288975 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.289646 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.290979 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.291580 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.291691 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:53.291701 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:53.291972 (+ 271us) server_negotiation.cc:299] Negotiation successful
0621 05:23:53.292020 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:23:53.292793 17878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1142ce01f9e84f61b8ec9e271cc1f98b" candidate_uuid: "11ba9033670a40f3b1dcd0cb865fa607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be235d3c549d44f0a9b4924326f225cb" is_pre_election: true
I20260621 05:23:53.292981 17878 raft_consensus.cc:2454] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11ba9033670a40f3b1dcd0cb865fa607 in term 0.
I20260621 05:23:53.293215 18012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1142ce01f9e84f61b8ec9e271cc1f98b" candidate_uuid: "11ba9033670a40f3b1dcd0cb865fa607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" is_pre_election: true
I20260621 05:23:53.293375 18012 raft_consensus.cc:2454] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11ba9033670a40f3b1dcd0cb865fa607 in term 0.
I20260621 05:23:53.293455 17680 leader_election.cc:302] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11ba9033670a40f3b1dcd0cb865fa607, be235d3c549d44f0a9b4924326f225cb; no voters:
I20260621 05:23:53.293598 18083 raft_consensus.cc:2781] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:23:53.293689 18083 raft_consensus.cc:482] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:23:53.293723 18083 raft_consensus.cc:3037] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:23:53.294571 18083 raft_consensus.cc:504] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } }
I20260621 05:23:53.294729 18083 leader_election.cc:288] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [CANDIDATE]: Term 1 election: Requested vote from peers d7bb06947ddb440fafc0bf38fa8b1037 (127.2.32.3:40135), be235d3c549d44f0a9b4924326f225cb (127.2.32.2:37849)
I20260621 05:23:53.294919 18012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1142ce01f9e84f61b8ec9e271cc1f98b" candidate_uuid: "11ba9033670a40f3b1dcd0cb865fa607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7bb06947ddb440fafc0bf38fa8b1037"
I20260621 05:23:53.294988 18012 raft_consensus.cc:3037] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:23:53.295657 18012 raft_consensus.cc:2454] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11ba9033670a40f3b1dcd0cb865fa607 in term 1.
I20260621 05:23:53.295773 17878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1142ce01f9e84f61b8ec9e271cc1f98b" candidate_uuid: "11ba9033670a40f3b1dcd0cb865fa607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be235d3c549d44f0a9b4924326f225cb"
I20260621 05:23:53.295820 17680 leader_election.cc:302] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11ba9033670a40f3b1dcd0cb865fa607, d7bb06947ddb440fafc0bf38fa8b1037; no voters:
I20260621 05:23:53.295861 17878 raft_consensus.cc:3037] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:23:53.295967 18083 raft_consensus.cc:2781] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:23:53.296149 18083 raft_consensus.cc:686] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 LEADER]: Becoming Leader. State: Replica: 11ba9033670a40f3b1dcd0cb865fa607, State: Running, Role: LEADER
I20260621 05:23:53.296257 18083 consensus_queue.cc:237] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } }
I20260621 05:23:53.296753 17878 raft_consensus.cc:2454] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11ba9033670a40f3b1dcd0cb865fa607 in term 1.
I20260621 05:23:53.297011 17598 catalog_manager.cc:5495] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11ba9033670a40f3b1dcd0cb865fa607 (127.2.32.1). New cstate: current_term: 1 leader_uuid: "11ba9033670a40f3b1dcd0cb865fa607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } health_report { overall_health: HEALTHY } } }
I20260621 05:23:53.318497 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:53.316052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44768 (local address 127.2.32.1:36527)
0621 05:23:53.316148 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0621 05:23:53.316150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:23:53.316161 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:23:53.316202 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:23:53.316203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:23:53.316227 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:23:53.316335 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:23:53.316468 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.316956 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.317721 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:23:53.317930 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:23:53.318007 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:23:53.318176 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:23:53.318277 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:23:53.318373 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 05:23:53.318392 (+ 19us) server_negotiation.cc:299] Negotiation successful
0621 05:23:53.318412 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 05:23:53.320688 18012 raft_consensus.cc:1261] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037 [term 1 FOLLOWER]: Refusing update from remote peer 11ba9033670a40f3b1dcd0cb865fa607: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:23:53.320688 17878 raft_consensus.cc:1261] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb [term 1 FOLLOWER]: Refusing update from remote peer 11ba9033670a40f3b1dcd0cb865fa607: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:23:53.321161 18085 consensus_queue.cc:1035] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:23:53.321369 18083 consensus_queue.cc:1035] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:23:53.327677 17599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:23:53.327818 17599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:23:53.328733 17599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1: Sending DeleteTablet for 3 replicas of tablet 1142ce01f9e84f61b8ec9e271cc1f98b
I20260621 05:23:53.329116 17724 tablet_service.cc:1514] Processing DeleteTablet for tablet 1142ce01f9e84f61b8ec9e271cc1f98b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:23:53 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:44758
I20260621 05:23:53.329210 17992 tablet_service.cc:1514] Processing DeleteTablet for tablet 1142ce01f9e84f61b8ec9e271cc1f98b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:23:53 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:60236
I20260621 05:23:53.329205 17858 tablet_service.cc:1514] Processing DeleteTablet for tablet 1142ce01f9e84f61b8ec9e271cc1f98b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:23:53 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:33092
I20260621 05:23:53.329371 18098 tablet_replica.cc:330] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607: stopping tablet replica
I20260621 05:23:53.329461 18098 raft_consensus.cc:2229] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:23:53.329520 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 17661
I20260621 05:23:53.329612 18098 raft_consensus.cc:2258] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260621 05:23:53.335074 17583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.32.1:36527 (error 108)
I20260621 05:23:53.335234 18099 tablet_replica.cc:330] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037: stopping tablet replica
I20260621 05:23:53.335317 18100 tablet_replica.cc:330] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb: stopping tablet replica
I20260621 05:23:53.335354 18099 raft_consensus.cc:2229] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:23:53.335474 18099 raft_consensus.cc:2258] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:23:53.335633 18069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:53.335520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:36527 (local address 127.0.0.1:44774)
0621 05:23:53.335566 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:53.335595 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.1:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260621 05:23:53.335716 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 17795
W20260621 05:23:53.335759 17583 catalog_manager.cc:4584] TS 11ba9033670a40f3b1dcd0cb865fa607 (127.2.32.1:36527): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1142ce01f9e84f61b8ec9e271cc1f98b: Network error: Client connection negotiation failed: client connection to 127.2.32.1:36527: connect: Connection refused (error 111)
I20260621 05:23:53.336910 18099 ts_tablet_manager.cc:1905] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:23:53.338963 18099 ts_tablet_manager.cc:1918] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:23:53.339044 18099 log.cc:1196] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/1142ce01f9e84f61b8ec9e271cc1f98b
I20260621 05:23:53.339350 18099 ts_tablet_manager.cc:1939] T 1142ce01f9e84f61b8ec9e271cc1f98b P d7bb06947ddb440fafc0bf38fa8b1037: Deleting consensus metadata
I20260621 05:23:53.339825 17586 catalog_manager.cc:4841] TS d7bb06947ddb440fafc0bf38fa8b1037 (127.2.32.3:40135): tablet 1142ce01f9e84f61b8ec9e271cc1f98b (table test-table [id=afdd56bd700445a4a0ae385416c8edc2]) successfully deleted
I20260621 05:23:53.341792 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 17929
I20260621 05:23:53.341866 18069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:23:53.341734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:37849 (local address 127.0.0.1:33098)
0621 05:23:53.341798 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0621 05:23:53.341830 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.2:37849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260621 05:23:53.346707 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 17567
I20260621 05:24:13.353425 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:40701
--webserver_interface=127.2.32.62
--webserver_port=43529
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:40701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:24:13.444425 18102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:13.444653 18102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:13.444690 18102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:13.445811 18102 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 05:24:13.445878 18102 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 05:24:13.445902 18102 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 05:24:13.445951 18102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:24:13.445981 18102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:13.446008 18102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:13.446033 18102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:24:13.446059 18102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:24:13.447566 18102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:40701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:40701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=43529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:13.447820 18102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:13.448037 18102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:13.457129 18108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:13.457125 18109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:13.457485 18102 server_base.cc:1034] running on GCE node
W20260621 05:24:13.457252 18111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:13.457922 18102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:13.458297 18102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:13.459546 18102 hybrid_clock.cc:648] HybridClock initialized: now 1782019453459511 us; error 42 us; skew 500 ppm
Jun 21 05:24:13 dist-test-slave-b357 krb5kdc[17554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:13.462752 18102 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:24:13.463490 18102 webserver.cc:460] Webserver started at http://127.2.32.62:43529/ using document root <none> and password file <none>
I20260621 05:24:13.463788 18102 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:13.463902 18102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:13.465690 18102 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:24:13.466614 18118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:13.466882 18102 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:13.467023 18102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "cb4f712f75ca4b649cf74281f34999e1"
format_stamp: "Formatted at 2026-06-21 05:23:52 on dist-test-slave-b357"
server_key: "ae9ec513fcf2b6d2a85e412b02e71fa9"
server_key_iv: "2a08e8ce492227228a9f81f632e2df00"
server_key_version: "encryptionkey@0"
I20260621 05:24:13.467552 18102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:13.485286 18102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:13.488287 18102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:13.488497 18102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:13.492815 18102 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:40701
I20260621 05:24:13.492908 18170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:40701 every 8 connection(s)
I20260621 05:24:13.493257 18102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:24:13.494637 18171 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:24:13.495381 18171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1: Bootstrap starting.
I20260621 05:24:13.499217 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 18102
I20260621 05:24:13.499581 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:36527
--local_ip_for_outbound_sockets=127.2.32.1
--tserver_master_addrs=127.2.32.62:40701
--webserver_port=38945
--webserver_interface=127.2.32.1
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:24:13.504200 18171 log.cc:824] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:13.506098 18171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:24:13.506402 18171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1: Bootstrap complete.
I20260621 05:24:13.508733 18171 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f712f75ca4b649cf74281f34999e1" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40701 } }
I20260621 05:24:13.509078 18171 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb4f712f75ca4b649cf74281f34999e1, State: Initialized, Role: FOLLOWER
I20260621 05:24:13.509245 18171 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f712f75ca4b649cf74281f34999e1" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40701 } }
I20260621 05:24:13.509331 18171 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:24:13.509394 18171 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:24:13.509442 18171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 1 FOLLOWER]: Advancing to term 2
I20260621 05:24:13.510521 18171 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f712f75ca4b649cf74281f34999e1" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40701 } }
I20260621 05:24:13.510639 18171 leader_election.cc:302] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb4f712f75ca4b649cf74281f34999e1; no voters:
I20260621 05:24:13.510830 18171 leader_election.cc:288] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 05:24:13.510864 18175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 05:24:13.510983 18175 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 2 LEADER]: Becoming Leader. State: Replica: cb4f712f75ca4b649cf74281f34999e1, State: Running, Role: LEADER
I20260621 05:24:13.511101 18171 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:24:13.511336 18175 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f712f75ca4b649cf74281f34999e1" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40701 } }
I20260621 05:24:13.511642 18175 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb4f712f75ca4b649cf74281f34999e1. Latest consensus state: current_term: 2 leader_uuid: "cb4f712f75ca4b649cf74281f34999e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f712f75ca4b649cf74281f34999e1" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40701 } } }
I20260621 05:24:13.511749 18175 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [sys.catalog]: This master's current role is: LEADER
I20260621 05:24:13.511647 18176 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cb4f712f75ca4b649cf74281f34999e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f712f75ca4b649cf74281f34999e1" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40701 } } }
I20260621 05:24:13.511978 18176 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [sys.catalog]: This master's current role is: LEADER
I20260621 05:24:13.512471 18184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:24:13.513417 18184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:24:13.513523 18184 catalog_manager.cc:1259] Loaded cluster ID: 0c656a092a7044f6b9b98ad26355ee4a
I20260621 05:24:13.513557 18184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:24:13.514779 18184 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:24:13.514943 18184 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1: Loaded TSK: 0
I20260621 05:24:13.535437 18184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1: Generated new TSK 1
I20260621 05:24:13.535540 18184 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 05:24:13.594283 18173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:13.594491 18173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:13.594514 18173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:13.595721 18173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:13.595763 18173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:13.595789 18173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:24:13.597205 18173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:36527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=38945
--tserver_master_addrs=127.2.32.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:13.597441 18173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:13.597693 18173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:13.602929 18198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:13.603073 18199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:13.603081 18173 server_base.cc:1034] running on GCE node
W20260621 05:24:13.603423 18201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:13.603731 18173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:13.604113 18173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:13.605314 18173 hybrid_clock.cc:648] HybridClock initialized: now 1782019453605276 us; error 52 us; skew 500 ppm
Jun 21 05:24:13 dist-test-slave-b357 krb5kdc[17554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:13.608783 18173 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:24:13.609534 18173 webserver.cc:460] Webserver started at http://127.2.32.1:38945/ using document root <none> and password file <none>
I20260621 05:24:13.609826 18173 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:13.610114 18173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:13.611707 18173 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:24:13.612504 18208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:13.612744 18173 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:24:13.612838 18173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "11ba9033670a40f3b1dcd0cb865fa607"
format_stamp: "Formatted at 2026-06-21 05:23:52 on dist-test-slave-b357"
server_key: "01c6bdf916ebb244e1e9bb0f1d90f54f"
server_key_iv: "f5380ecc90cbc280d3b1e531e7ddf343"
server_key_version: "encryptionkey@0"
I20260621 05:24:13.613196 18173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:13.627709 18173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:13.630163 18173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:13.630332 18173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:13.630600 18173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:13.631088 18215 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 05:24:13.633139 18173 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 05:24:13.633221 18173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260621 05:24:13.633255 18173 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 05:24:13.634074 18173 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 05:24:13.634124 18173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260621 05:24:13.634166 18215 tablet_bootstrap.cc:492] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607: Bootstrap starting.
I20260621 05:24:13.642562 18173 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:36527
I20260621 05:24:13.642812 18322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:36527 every 8 connection(s)
I20260621 05:24:13.643121 18173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:24:13.645795 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 18173
I20260621 05:24:13.646144 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:37849
--local_ip_for_outbound_sockets=127.2.32.2
--tserver_master_addrs=127.2.32.62:40701
--webserver_port=33649
--webserver_interface=127.2.32.2
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:24:13.646548 18215 log.cc:824] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:13.648844 18215 tablet_bootstrap.cc:492] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:24:13.649147 18215 tablet_bootstrap.cc:492] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607: Bootstrap complete.
I20260621 05:24:13.650264 18215 ts_tablet_manager.cc:1397] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20260621 05:24:13.651549 18215 raft_consensus.cc:348] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } }
I20260621 05:24:13.651803 18215 raft_consensus.cc:729] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11ba9033670a40f3b1dcd0cb865fa607, State: Initialized, Role: FOLLOWER
I20260621 05:24:13.651937 18215 consensus_queue.cc:260] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } }
I20260621 05:24:13.652245 18215 ts_tablet_manager.cc:1428] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607: Time spent starting tablet: real 0.002s user 0.003s sys 0.002s
Jun 21 05:24:13 dist-test-slave-b357 krb5kdc[17554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:24:13.656613 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:13.645435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:50503 (local address 127.2.32.62:40701)
0621 05:24:13.645614 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:13.645623 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:13.645659 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:24:13.645792 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:13.645798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:13.645865 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:13.646844 (+ 979us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:13.646947 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.648302 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.649594 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.650266 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.650424 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:13.653078 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:13.653082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:13.653082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:13.653166 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:13.654831 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:13.655349 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:13.655352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:13.655352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:13.655447 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:13.655700 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:13.655701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:13.655702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:13.655925 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:13.656014 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:13.656309 (+ 295us) server_negotiation.cc:299] Negotiation successful
0621 05:24:13.656435 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":96,"threads_started":1}
I20260621 05:24:13.656615 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:13.644148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.1:50503)
0621 05:24:13.644429 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:13.644499 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:13.645231 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:13.646075 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:13.646079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:13.646491 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:13.646780 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:13.646787 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.648461 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.648463 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:13.649518 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:13.649520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.650441 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.650443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:13.650599 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:13.651266 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:24:13.651277 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:13.652969 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:13.654949 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:13.654956 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:13.654958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:13.655254 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:13.655519 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:13.655520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:13.655521 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:13.655631 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:13.656006 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:13.656012 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:13.656142 (+ 130us) client_negotiation.cc:769] Sending connection context
0621 05:24:13.656245 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 05:24:13.656427 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":106,"threads_started":1}
I20260621 05:24:13.657567 18323 heartbeater.cc:347] Connected to a master server at 127.2.32.62:40701
I20260621 05:24:13.657673 18323 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:13.657887 18323 heartbeater.cc:511] Master 127.2.32.62:40701 requested a full tablet report, sending...
I20260621 05:24:13.658586 18135 ts_manager.cc:194] Registered new tserver with Master: 11ba9033670a40f3b1dcd0cb865fa607 (127.2.32.1:36527)
I20260621 05:24:13.659624 18135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:50503
I20260621 05:24:13.660813 18323 heartbeater.cc:503] Master 127.2.32.62:40701 was elected leader, sending a full tablet report...
Jun 21 05:24:13 dist-test-slave-b357 krb5kdc[17554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for kudu/127.2.32.1@KRBTEST.COM
I20260621 05:24:13.667883 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:13.659936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53462 (local address 127.2.32.1:36527)
0621 05:24:13.660108 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:13.660112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:13.660341 (+ 229us) server_negotiation.cc:407] Connection header received
0621 05:24:13.660375 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:13.660377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:13.660411 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:13.660468 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:13.661082 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.661894 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.662698 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.662994 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.663075 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:13.665061 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:13.665068 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:13.665069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:13.665075 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:13.666576 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:13.667000 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:13.667002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:13.667003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:13.667059 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:13.667310 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:13.667311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:13.667311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:13.667440 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:13.667529 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:13.667715 (+ 186us) server_negotiation.cc:299] Negotiation successful
0621 05:24:13.667789 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":111,"threads_started":1}
I20260621 05:24:13.667883 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:13.659179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:36527 (local address 127.0.0.1:53462)
0621 05:24:13.660138 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:13.660171 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:13.660265 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:13.660496 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:13.660501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:13.660934 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:13.660993 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:13.660995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.661980 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.661982 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:13.662594 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:13.662596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.663071 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.663075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:13.663172 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:13.663668 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:24:13.663674 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:13.664970 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:13.666679 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:13.666686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:13.666688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:13.666877 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:13.667167 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:13.667168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:13.667168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:13.667228 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:13.667519 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:13.667525 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:13.667575 (+ 50us) client_negotiation.cc:769] Sending connection context
0621 05:24:13.667667 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 05:24:13.667790 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"thread_start_us":90,"threads_started":1}
I20260621 05:24:13.668289 18257 tablet_service.cc:1514] Processing DeleteTablet for tablet 1142ce01f9e84f61b8ec9e271cc1f98b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:23:53 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:53462
I20260621 05:24:13.668771 18332 tablet_replica.cc:330] stopping tablet replica
I20260621 05:24:13.668862 18332 raft_consensus.cc:2229] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:24:13.668959 18332 raft_consensus.cc:2258] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:24:13.669272 18332 ts_tablet_manager.cc:1905] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:24:13.671446 18332 ts_tablet_manager.cc:1918] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:24:13.671505 18332 log.cc:1196] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/wal/wals/1142ce01f9e84f61b8ec9e271cc1f98b
I20260621 05:24:13.671728 18332 ts_tablet_manager.cc:1939] T 1142ce01f9e84f61b8ec9e271cc1f98b P 11ba9033670a40f3b1dcd0cb865fa607: Deleting consensus metadata
I20260621 05:24:13.672186 18119 catalog_manager.cc:4841] TS 11ba9033670a40f3b1dcd0cb865fa607 (127.2.32.1:36527): tablet 1142ce01f9e84f61b8ec9e271cc1f98b (table test-table [id=afdd56bd700445a4a0ae385416c8edc2]) successfully deleted
W20260621 05:24:13.732573 18328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:13.732808 18328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:13.732843 18328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:13.734105 18328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:13.734191 18328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:13.734243 18328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:24:13.735750 18328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:37849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=33649
--tserver_master_addrs=127.2.32.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:13.736016 18328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:13.736302 18328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:13.741581 18337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:13.741575 18340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:13.741632 18338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:13.741964 18328 server_base.cc:1034] running on GCE node
I20260621 05:24:13.742161 18328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:13.742386 18328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:13.743580 18328 hybrid_clock.cc:648] HybridClock initialized: now 1782019453743569 us; error 37 us; skew 500 ppm
Jun 21 05:24:13 dist-test-slave-b357 krb5kdc[17554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:13.746455 18328 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:24:13.747201 18328 webserver.cc:460] Webserver started at http://127.2.32.2:33649/ using document root <none> and password file <none>
I20260621 05:24:13.747470 18328 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:13.747571 18328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:13.749197 18328 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:24:13.750003 18347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:13.750211 18328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:13.750325 18328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "be235d3c549d44f0a9b4924326f225cb"
format_stamp: "Formatted at 2026-06-21 05:23:52 on dist-test-slave-b357"
server_key: "5483910a424db3844a921432c28d7377"
server_key_iv: "bafd5d9517524dc182486b91aa4831ee"
server_key_version: "encryptionkey@0"
I20260621 05:24:13.750727 18328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:13.762908 18328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:13.765676 18328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:13.765914 18328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:13.766214 18328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:13.766727 18354 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 05:24:13.768272 18328 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 05:24:13.768370 18328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260621 05:24:13.768426 18328 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 05:24:13.769272 18328 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 05:24:13.769332 18328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260621 05:24:13.769488 18354 tablet_bootstrap.cc:492] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb: Bootstrap starting.
I20260621 05:24:13.776989 18328 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:37849
I20260621 05:24:13.777029 18461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:37849 every 8 connection(s)
I20260621 05:24:13.777398 18328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:24:13.779393 18354 log.cc:824] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:13.781920 18354 tablet_bootstrap.cc:492] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:24:13.782217 18354 tablet_bootstrap.cc:492] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb: Bootstrap complete.
I20260621 05:24:13.783020 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 18328
I20260621 05:24:13.783344 18354 ts_tablet_manager.cc:1397] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.006s
I20260621 05:24:13.783478 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:40135
--local_ip_for_outbound_sockets=127.2.32.3
--tserver_master_addrs=127.2.32.62:40701
--webserver_port=37085
--webserver_interface=127.2.32.3
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:24:13.784710 18354 raft_consensus.cc:348] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } }
I20260621 05:24:13.784992 18354 raft_consensus.cc:729] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: be235d3c549d44f0a9b4924326f225cb, State: Initialized, Role: FOLLOWER
I20260621 05:24:13.785162 18354 consensus_queue.cc:260] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } }
Jun 21 05:24:13 dist-test-slave-b357 krb5kdc[17554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:24:13.785449 18354 ts_tablet_manager.cc:1428] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260621 05:24:13.789865 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:13.778152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.2:59101)
0621 05:24:13.778383 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:13.778434 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:13.779176 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:13.779550 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:13.779555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:13.779851 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:13.780046 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:13.780050 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.781042 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.781043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:13.782295 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:13.782298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.782678 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.782679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:13.782853 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:13.783801 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:24:13.783810 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:13.785818 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:13.788474 (+ 2656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:13.788480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:13.788481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:13.788734 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:13.788961 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:13.788961 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:13.788962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:13.789057 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:13.789389 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:13.789395 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:13.789510 (+ 115us) client_negotiation.cc:769] Sending connection context
0621 05:24:13.789606 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 05:24:13.789729 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":114,"threads_started":1}
I20260621 05:24:13.789969 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:13.778204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:59101 (local address 127.2.32.62:40701)
0621 05:24:13.778284 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:13.778286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:13.778543 (+ 257us) server_negotiation.cc:407] Connection header received
0621 05:24:13.779341 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:13.779342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:13.779365 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:13.779467 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:13.780198 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.780911 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.782374 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.782584 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.782658 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:13.785951 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:13.785953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:13.785954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:13.785967 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:13.788355 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:13.788817 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:13.788818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:13.788819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:13.788889 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:13.789122 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:13.789123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:13.789124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:13.789283 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:13.789362 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:13.789724 (+ 362us) server_negotiation.cc:299] Negotiation successful
0621 05:24:13.789838 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 05:24:13.790524 18462 heartbeater.cc:347] Connected to a master server at 127.2.32.62:40701
I20260621 05:24:13.790606 18462 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:13.790799 18462 heartbeater.cc:511] Master 127.2.32.62:40701 requested a full tablet report, sending...
I20260621 05:24:13.791352 18135 ts_manager.cc:194] Registered new tserver with Master: be235d3c549d44f0a9b4924326f225cb (127.2.32.2:37849)
I20260621 05:24:13.792016 18135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:59101
I20260621 05:24:13.792742 18462 heartbeater.cc:503] Master 127.2.32.62:40701 was elected leader, sending a full tablet report...
Jun 21 05:24:13 dist-test-slave-b357 krb5kdc[17554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for kudu/127.2.32.2@KRBTEST.COM
I20260621 05:24:13.799588 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:13.791773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:37849 (local address 127.0.0.1:54468)
0621 05:24:13.791822 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:13.791833 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:13.791874 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:13.792106 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:13.792107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:13.792271 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:13.792336 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:13.792339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.793720 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.793721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:13.794207 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:13.794208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.794888 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.794891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:13.794947 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:13.795427 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:24:13.795432 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:13.796589 (+ 1157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:13.798496 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:13.798496 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:13.798497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:13.798673 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:13.798889 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:13.798889 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:13.798890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:13.798944 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:13.799316 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:13.799317 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:13.799368 (+ 51us) client_negotiation.cc:769] Sending connection context
0621 05:24:13.799444 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 05:24:13.799524 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260621 05:24:13.799921 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:13.791824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54468 (local address 127.2.32.2:37849)
0621 05:24:13.791974 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:13.791978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:13.791985 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:24:13.792009 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:13.792011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:13.792021 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:13.792103 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:13.792424 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.793618 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.794292 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.794796 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.794880 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:13.796668 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:13.796674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:13.796675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:13.796681 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:13.798400 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:13.798755 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:13.798758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:13.798758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:13.798819 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:13.799042 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:13.799043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:13.799043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:13.799222 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:13.799303 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:13.799623 (+ 320us) server_negotiation.cc:299] Negotiation successful
0621 05:24:13.799798 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
I20260621 05:24:13.800482 18396 tablet_service.cc:1514] Processing DeleteTablet for tablet 1142ce01f9e84f61b8ec9e271cc1f98b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:23:53 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:54468
I20260621 05:24:13.801229 18469 tablet_replica.cc:330] stopping tablet replica
I20260621 05:24:13.801357 18469 raft_consensus.cc:2229] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:24:13.801434 18469 raft_consensus.cc:2258] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:24:13.801721 18469 ts_tablet_manager.cc:1905] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:24:13.803917 18469 ts_tablet_manager.cc:1918] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:24:13.804019 18469 log.cc:1196] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/1142ce01f9e84f61b8ec9e271cc1f98b
I20260621 05:24:13.804361 18469 ts_tablet_manager.cc:1939] T 1142ce01f9e84f61b8ec9e271cc1f98b P be235d3c549d44f0a9b4924326f225cb: Deleting consensus metadata
I20260621 05:24:13.804968 18122 catalog_manager.cc:4841] TS be235d3c549d44f0a9b4924326f225cb (127.2.32.2:37849): tablet 1142ce01f9e84f61b8ec9e271cc1f98b (table test-table [id=afdd56bd700445a4a0ae385416c8edc2]) successfully deleted
W20260621 05:24:13.873108 18466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:13.873330 18466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:13.873368 18466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:13.874492 18466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:13.874555 18466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:13.874600 18466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:24:13.876471 18466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:40135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=37085
--tserver_master_addrs=127.2.32.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:13.876700 18466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:13.876919 18466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:13.882213 18475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:13.882190 18474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:13.882592 18477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:13.882620 18466 server_base.cc:1034] running on GCE node
I20260621 05:24:13.882884 18466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:13.883200 18466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:13.884404 18466 hybrid_clock.cc:648] HybridClock initialized: now 1782019453884378 us; error 42 us; skew 500 ppm
Jun 21 05:24:13 dist-test-slave-b357 krb5kdc[17554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:13.887095 18466 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:24:13.887598 18466 webserver.cc:460] Webserver started at http://127.2.32.3:37085/ using document root <none> and password file <none>
I20260621 05:24:13.887851 18466 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:13.887921 18466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:13.889384 18466 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260621 05:24:13.890167 18484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:13.890434 18466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:24:13.890512 18466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "d7bb06947ddb440fafc0bf38fa8b1037"
format_stamp: "Formatted at 2026-06-21 05:23:53 on dist-test-slave-b357"
server_key: "b4bce4edab6920e49a556c9516153204"
server_key_iv: "76cad499582e59b19331684fa617939a"
server_key_version: "encryptionkey@0"
I20260621 05:24:13.890861 18466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:13.904559 18466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:13.907516 18466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:13.907708 18466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:13.908027 18466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:13.908447 18466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:13.908519 18466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:13.908597 18466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:13.908631 18466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:13.916446 18466 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:40135
I20260621 05:24:13.916523 18597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:40135 every 8 connection(s)
I20260621 05:24:13.916937 18466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:24:13.920475 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 18466
Jun 21 05:24:13 dist-test-slave-b357 krb5kdc[17554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:24:13.928763 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:13.917756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:53021 (local address 127.2.32.62:40701)
0621 05:24:13.917824 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:13.917826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:13.918089 (+ 263us) server_negotiation.cc:407] Connection header received
0621 05:24:13.918641 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:13.918642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:13.918652 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:13.918714 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:13.919741 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.920280 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.921726 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.922000 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.922086 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:13.924419 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:13.924421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:13.924422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:13.924429 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:13.926513 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:13.927288 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:13.927289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:13.927290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:13.927354 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:13.927735 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:13.927736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:13.927737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:13.927891 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:13.927982 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:13.928519 (+ 537us) server_negotiation.cc:299] Negotiation successful
0621 05:24:13.928632 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 05:24:13.928982 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:13.920906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.2.32.62:40701)
0621 05:24:13.921764 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:13.921766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:13.921773 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:24:13.921798 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:13.921799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:13.921811 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:13.921871 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:13.922538 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.923000 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.923761 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.923956 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.924029 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:13.925327 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:13.925328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:13.925329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:13.925335 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:13.927487 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:13.928037 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:13.928038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:13.928038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:13.928120 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:13.928356 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:13.928357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:13.928357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:13.928509 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:13.928692 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:13.928774 (+ 82us) server_negotiation.cc:299] Negotiation successful
0621 05:24:13.928893 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":77,"threads_started":1}
I20260621 05:24:13.929003 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:13.917740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.3:53021)
0621 05:24:13.917928 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:13.917969 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:13.918498 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:13.918829 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:13.918833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:13.919460 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:13.919642 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:13.919646 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.920434 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.920435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:13.921608 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:13.921613 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.922123 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.922124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:13.922258 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:13.922854 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:24:13.922863 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:13.924321 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:13.926650 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:13.926661 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:13.926664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:13.927169 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:13.927434 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:13.927435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:13.927435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:13.927593 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:13.928096 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:13.928104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:13.928338 (+ 234us) client_negotiation.cc:769] Sending connection context
0621 05:24:13.928469 (+ 131us) client_negotiation.cc:240] Negotiation successful
0621 05:24:13.928705 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260621 05:24:13.929688 18598 heartbeater.cc:347] Connected to a master server at 127.2.32.62:40701
I20260621 05:24:13.929816 18598 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:13.930016 18598 heartbeater.cc:511] Master 127.2.32.62:40701 requested a full tablet report, sending...
I20260621 05:24:13.930454 18135 ts_manager.cc:194] Registered new tserver with Master: d7bb06947ddb440fafc0bf38fa8b1037 (127.2.32.3:40135)
I20260621 05:24:13.930836 18135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:53021
I20260621 05:24:13.941542 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:24:13.945415 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:13.942148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.2.32.62:40701)
0621 05:24:13.942248 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:13.942251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:13.942370 (+ 119us) server_negotiation.cc:407] Connection header received
0621 05:24:13.942515 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:13.942516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:13.942583 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:13.942676 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:24:13.942852 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.943546 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.944437 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.944686 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.944800 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:13.945000 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:24:13.945139 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:24:13.945250 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:13.945271 (+ 21us) server_negotiation.cc:299] Negotiation successful
0621 05:24:13.945301 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 05:24:13.945837 18135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:24:13.948251 18135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:24:13.951077 18257 tablet_service.cc:1467] Processing CreateTablet for tablet 88535e932a6045c396770c09981bb6fe (DEFAULT_TABLE table=test-table [id=36d632ce6ccd49d0b13cf118999bf7e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:13.951076 18396 tablet_service.cc:1467] Processing CreateTablet for tablet 88535e932a6045c396770c09981bb6fe (DEFAULT_TABLE table=test-table [id=36d632ce6ccd49d0b13cf118999bf7e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:13.951359 18257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88535e932a6045c396770c09981bb6fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:13.951380 18396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88535e932a6045c396770c09981bb6fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:13.952737 18215 tablet_bootstrap.cc:492] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607: Bootstrap starting.
I20260621 05:24:13.953100 18215 tablet_bootstrap.cc:654] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:13.953312 18354 tablet_bootstrap.cc:492] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb: Bootstrap starting.
I20260621 05:24:13.953713 18215 tablet_bootstrap.cc:492] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607: No bootstrap required, opened a new log
I20260621 05:24:13.953755 18215 ts_tablet_manager.cc:1397] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260621 05:24:13.953851 18354 tablet_bootstrap.cc:654] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:13.954093 18215 raft_consensus.cc:348] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } }
I20260621 05:24:13.954160 18215 raft_consensus.cc:374] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:13.954193 18215 raft_consensus.cc:729] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11ba9033670a40f3b1dcd0cb865fa607, State: Initialized, Role: FOLLOWER
I20260621 05:24:13.954241 18215 consensus_queue.cc:260] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } }
I20260621 05:24:13.954311 18215 ts_tablet_manager.cc:1428] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260621 05:24:13.954912 18354 tablet_bootstrap.cc:492] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb: No bootstrap required, opened a new log
I20260621 05:24:13.954969 18354 ts_tablet_manager.cc:1397] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:24:13.955121 18354 raft_consensus.cc:348] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } }
I20260621 05:24:13.955230 18354 raft_consensus.cc:374] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:13.955288 18354 raft_consensus.cc:729] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be235d3c549d44f0a9b4924326f225cb, State: Initialized, Role: FOLLOWER
I20260621 05:24:13.955339 18354 consensus_queue.cc:260] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } }
I20260621 05:24:13.955435 18354 ts_tablet_manager.cc:1428] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260621 05:24:13.956848 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:13.950679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:40135 (local address 127.0.0.1:57574)
0621 05:24:13.950794 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:13.950814 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:13.950855 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:13.953046 (+ 2191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:13.953048 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:13.953050 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:13.953105 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:13.953107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.954562 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.954563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:13.955779 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:13.955782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.956595 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.956596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:13.956656 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:13.956675 (+ 19us) client_negotiation.cc:769] Sending connection context
0621 05:24:13.956733 (+ 58us) client_negotiation.cc:240] Negotiation successful
0621 05:24:13.956777 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260621 05:24:13.956861 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:13.950775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.2.32.3:40135)
0621 05:24:13.952833 (+ 2058us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:13.952838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:13.952850 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:24:13.952897 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:13.952899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:13.952915 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:13.953184 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:13.953271 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.954347 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.955923 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:13.956496 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:13.956593 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:13.956653 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:13.956749 (+ 96us) server_negotiation.cc:299] Negotiation successful
0621 05:24:13.956783 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2024,"thread_start_us":126,"threads_started":1}
I20260621 05:24:13.957538 18532 tablet_service.cc:1467] Processing CreateTablet for tablet 88535e932a6045c396770c09981bb6fe (DEFAULT_TABLE table=test-table [id=36d632ce6ccd49d0b13cf118999bf7e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:13.957794 18532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88535e932a6045c396770c09981bb6fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:13.960546 18607 tablet_bootstrap.cc:492] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037: Bootstrap starting.
I20260621 05:24:13.961576 18607 tablet_bootstrap.cc:654] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:13.962466 18607 log.cc:824] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:13.963471 18607 tablet_bootstrap.cc:492] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037: No bootstrap required, opened a new log
I20260621 05:24:13.963573 18607 ts_tablet_manager.cc:1397] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:24:13.965409 18607 raft_consensus.cc:348] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } }
I20260621 05:24:13.965588 18607 raft_consensus.cc:374] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:13.965627 18607 raft_consensus.cc:729] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7bb06947ddb440fafc0bf38fa8b1037, State: Initialized, Role: FOLLOWER
I20260621 05:24:13.965765 18607 consensus_queue.cc:260] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } }
I20260621 05:24:13.966109 18607 ts_tablet_manager.cc:1428] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260621 05:24:13.966138 18598 heartbeater.cc:503] Master 127.2.32.62:40701 was elected leader, sending a full tablet report...
W20260621 05:24:14.028095 18463 tablet.cc:2369] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:24:14.130198 18467 raft_consensus.cc:482] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:24:14.130409 18467 raft_consensus.cc:504] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } }
I20260621 05:24:14.130774 18467 leader_election.cc:288] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7bb06947ddb440fafc0bf38fa8b1037 (127.2.32.3:40135), 11ba9033670a40f3b1dcd0cb865fa607 (127.2.32.1:36527)
I20260621 05:24:14.134938 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:14.131173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:38377 (local address 127.2.32.3:40135)
0621 05:24:14.131220 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:14.131222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:14.131355 (+ 133us) server_negotiation.cc:407] Connection header received
0621 05:24:14.131383 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:14.131384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:14.131396 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:14.131450 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:14.131637 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:14.132376 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:14.133938 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:14.134500 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:14.134580 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:14.134593 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:14.134831 (+ 238us) server_negotiation.cc:299] Negotiation successful
0621 05:24:14.134868 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:24:14.134934 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:14.131106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:37837 (local address 127.2.32.1:36527)
0621 05:24:14.131182 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:14.131185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:14.131196 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:24:14.131290 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:14.131291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:14.131339 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:14.131423 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:14.131634 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:14.132376 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:14.133938 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:14.134556 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:14.134619 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:14.134682 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:14.134824 (+ 142us) server_negotiation.cc:299] Negotiation successful
0621 05:24:14.134866 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 05:24:14.134963 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:14.131004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:36527 (local address 127.2.32.2:37837)
0621 05:24:14.131070 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:14.131086 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:14.131193 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:14.131464 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:14.131465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:14.131467 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:14.131561 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:14.131563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:14.132513 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:14.132514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:14.133811 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:14.133814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:14.134640 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:14.134642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:14.134721 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:14.134736 (+ 15us) client_negotiation.cc:769] Sending connection context
0621 05:24:14.134809 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 05:24:14.134857 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":114}
I20260621 05:24:14.134969 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:14.131008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:40135 (local address 127.2.32.2:38377)
0621 05:24:14.131265 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:14.131275 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:14.131328 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:14.131470 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:14.131470 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:14.131471 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:14.131550 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:14.131552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:14.132513 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:14.132514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:14.133811 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:14.133814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:14.134652 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:14.134653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:14.134716 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:14.134737 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 05:24:14.134814 (+ 77us) client_negotiation.cc:240] Negotiation successful
0621 05:24:14.134853 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":100,"thread_start_us":131,"threads_started":1}
I20260621 05:24:14.135401 18277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88535e932a6045c396770c09981bb6fe" candidate_uuid: "be235d3c549d44f0a9b4924326f225cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11ba9033670a40f3b1dcd0cb865fa607" is_pre_election: true
I20260621 05:24:14.135445 18552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88535e932a6045c396770c09981bb6fe" candidate_uuid: "be235d3c549d44f0a9b4924326f225cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" is_pre_election: true
I20260621 05:24:14.135583 18277 raft_consensus.cc:2454] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be235d3c549d44f0a9b4924326f225cb in term 0.
I20260621 05:24:14.135583 18552 raft_consensus.cc:2454] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be235d3c549d44f0a9b4924326f225cb in term 0.
I20260621 05:24:14.135783 18348 leader_election.cc:302] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11ba9033670a40f3b1dcd0cb865fa607, be235d3c549d44f0a9b4924326f225cb; no voters:
I20260621 05:24:14.135958 18467 raft_consensus.cc:2781] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:24:14.136011 18467 raft_consensus.cc:482] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:24:14.136029 18467 raft_consensus.cc:3037] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:14.136842 18467 raft_consensus.cc:504] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } }
I20260621 05:24:14.136989 18467 leader_election.cc:288] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [CANDIDATE]: Term 1 election: Requested vote from peers d7bb06947ddb440fafc0bf38fa8b1037 (127.2.32.3:40135), 11ba9033670a40f3b1dcd0cb865fa607 (127.2.32.1:36527)
I20260621 05:24:14.137112 18552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88535e932a6045c396770c09981bb6fe" candidate_uuid: "be235d3c549d44f0a9b4924326f225cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7bb06947ddb440fafc0bf38fa8b1037"
I20260621 05:24:14.137131 18277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88535e932a6045c396770c09981bb6fe" candidate_uuid: "be235d3c549d44f0a9b4924326f225cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11ba9033670a40f3b1dcd0cb865fa607"
I20260621 05:24:14.137187 18552 raft_consensus.cc:3037] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:14.137192 18277 raft_consensus.cc:3037] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:14.137971 18277 raft_consensus.cc:2454] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be235d3c549d44f0a9b4924326f225cb in term 1.
I20260621 05:24:14.138029 18552 raft_consensus.cc:2454] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be235d3c549d44f0a9b4924326f225cb in term 1.
I20260621 05:24:14.138147 18348 leader_election.cc:302] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11ba9033670a40f3b1dcd0cb865fa607, be235d3c549d44f0a9b4924326f225cb; no voters:
I20260621 05:24:14.138248 18467 raft_consensus.cc:2781] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:24:14.138409 18467 raft_consensus.cc:686] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [term 1 LEADER]: Becoming Leader. State: Replica: be235d3c549d44f0a9b4924326f225cb, State: Running, Role: LEADER
I20260621 05:24:14.138499 18467 consensus_queue.cc:237] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } }
I20260621 05:24:14.139273 18134 catalog_manager.cc:5495] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb reported cstate change: term changed from 0 to 1, leader changed from <none> to be235d3c549d44f0a9b4924326f225cb (127.2.32.2). New cstate: current_term: 1 leader_uuid: "be235d3c549d44f0a9b4924326f225cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } health_report { overall_health: HEALTHY } } }
W20260621 05:24:14.144198 18324 tablet.cc:2369] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:24:14.167694 18599 tablet.cc:2369] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:24:14.186726 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:14.184312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54478 (local address 127.2.32.2:37849)
0621 05:24:14.184380 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:14.184382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:14.184396 (+ 14us) server_negotiation.cc:407] Connection header received
0621 05:24:14.184510 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:14.184511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:14.184539 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:14.184619 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:24:14.184749 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:14.185254 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:14.185991 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:14.186206 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:14.186281 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:14.186441 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:24:14.186497 (+ 56us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0621 05:24:14.186644 (+ 147us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54478: token expired
Metrics: {"server-negotiator.queue_time_us":41}
W20260621 05:24:14.186794 18468 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54478: token expired
I20260621 05:24:14.194310 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:14.187296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42194 (local address 127.2.32.62:40701)
0621 05:24:14.187372 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:14.187374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:14.187383 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:24:14.187414 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:14.187415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:14.187426 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:14.187528 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:14.188155 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:14.188715 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:14.189449 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:14.189688 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:14.189878 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:14.191305 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:14.191306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:14.191307 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:14.191315 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:14.193009 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:14.193412 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:14.193413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:14.193414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:14.193474 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:14.193679 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:14.193680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:14.193681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:14.193839 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:14.193946 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:14.194108 (+ 162us) server_negotiation.cc:299] Negotiation successful
0621 05:24:14.194207 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 05:24:14.197688 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:14.195508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54482 (local address 127.2.32.2:37849)
0621 05:24:14.195547 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:14.195553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:14.195561 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:24:14.195581 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:14.195581 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:14.195589 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:14.195643 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:24:14.195805 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:14.196250 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:14.197086 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:14.197283 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:14.197341 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:14.197453 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:24:14.197535 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:24:14.197577 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:14.197624 (+ 47us) server_negotiation.cc:299] Negotiation successful
0621 05:24:14.197638 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:24:14.199410 18552 raft_consensus.cc:1261] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037 [term 1 FOLLOWER]: Refusing update from remote peer be235d3c549d44f0a9b4924326f225cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:24:14.199707 18277 raft_consensus.cc:1261] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Refusing update from remote peer be235d3c549d44f0a9b4924326f225cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:24:14.199712 18467 consensus_queue.cc:1035] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:24:14.199973 18467 consensus_queue.cc:1035] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:24:14.201679 18614 mvcc.cc:205] Tried to move back new op lower bound from 7299151684398170112 to 7299151684151783424. Current Snapshot: MvccSnapshot[applied={T|T < 7299151684398170112}]
I20260621 05:24:14.203279 18616 mvcc.cc:205] Tried to move back new op lower bound from 7299151684398170112 to 7299151684151783424. Current Snapshot: MvccSnapshot[applied={T|T < 7299151684398170112}]
I20260621 05:24:14.204339 18134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:24:14.204468 18134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:24:14.205338 18134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1: Sending DeleteTablet for 3 replicas of tablet 88535e932a6045c396770c09981bb6fe
I20260621 05:24:14.205680 18396 tablet_service.cc:1514] Processing DeleteTablet for tablet 88535e932a6045c396770c09981bb6fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:24:14 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:54468
I20260621 05:24:14.205683 18532 tablet_service.cc:1514] Processing DeleteTablet for tablet 88535e932a6045c396770c09981bb6fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:24:14 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:57574
I20260621 05:24:14.205781 18469 tablet_replica.cc:330] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb: stopping tablet replica
I20260621 05:24:14.205799 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 18173
I20260621 05:24:14.205844 18469 raft_consensus.cc:2229] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:24:14.205931 18469 raft_consensus.cc:2258] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:24:14.206771 18469 ts_tablet_manager.cc:1905] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:24:14.207304 18623 tablet_replica.cc:330] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037: stopping tablet replica
I20260621 05:24:14.207422 18623 raft_consensus.cc:2229] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:24:14.207544 18623 raft_consensus.cc:2258] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:24:14.208165 18623 ts_tablet_manager.cc:1905] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:24:14.208700 18469 ts_tablet_manager.cc:1918] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:24:14.208755 18469 log.cc:1196] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/88535e932a6045c396770c09981bb6fe
I20260621 05:24:14.208987 18469 ts_tablet_manager.cc:1939] T 88535e932a6045c396770c09981bb6fe P be235d3c549d44f0a9b4924326f225cb: Deleting consensus metadata
I20260621 05:24:14.209504 18122 catalog_manager.cc:4841] TS be235d3c549d44f0a9b4924326f225cb (127.2.32.2:37849): tablet 88535e932a6045c396770c09981bb6fe (table test-table [id=36d632ce6ccd49d0b13cf118999bf7e9]) successfully deleted
I20260621 05:24:14.210314 18623 ts_tablet_manager.cc:1918] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:24:14.210407 18623 log.cc:1196] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/88535e932a6045c396770c09981bb6fe
I20260621 05:24:14.210788 18623 ts_tablet_manager.cc:1939] T 88535e932a6045c396770c09981bb6fe P d7bb06947ddb440fafc0bf38fa8b1037: Deleting consensus metadata
I20260621 05:24:14.211330 18122 catalog_manager.cc:4841] TS d7bb06947ddb440fafc0bf38fa8b1037 (127.2.32.3:40135): tablet 88535e932a6045c396770c09981bb6fe (table test-table [id=36d632ce6ccd49d0b13cf118999bf7e9]) successfully deleted
W20260621 05:24:14.212953 18119 connection.cc:535] client connection to 127.2.32.1:36527 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260621 05:24:14.213083 18119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 05:24:14.213277 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 18328
I20260621 05:24:14.213590 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:14.213470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:36527 (local address 127.0.0.1:53468)
0621 05:24:14.213518 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:14.213552 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.1:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260621 05:24:14.213699 18119 catalog_manager.cc:4584] TS 11ba9033670a40f3b1dcd0cb865fa607 (127.2.32.1:36527): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 88535e932a6045c396770c09981bb6fe: Network error: Client connection negotiation failed: client connection to 127.2.32.1:36527: connect: Connection refused (error 111)
I20260621 05:24:14.219327 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 18466
I20260621 05:24:14.223888 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 18102
I20260621 05:24:34.229679 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:40701
--webserver_interface=127.2.32.62
--webserver_port=43529
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:40701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:24:34.329764 18625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:34.330093 18625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:34.330125 18625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:34.331725 18625 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 05:24:34.331775 18625 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 05:24:34.331795 18625 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 05:24:34.331808 18625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:24:34.331826 18625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:34.331840 18625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:34.331853 18625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:24:34.331866 18625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:24:34.333600 18625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:40701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:40701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=43529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:34.333886 18625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:34.334115 18625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:34.339741 18630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:34.339764 18633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:34.339735 18631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:34.340224 18625 server_base.cc:1034] running on GCE node
I20260621 05:24:34.340701 18625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:34.341506 18625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:34.342739 18625 hybrid_clock.cc:648] HybridClock initialized: now 1782019474342715 us; error 42 us; skew 500 ppm
Jun 21 05:24:34 dist-test-slave-b357 krb5kdc[17554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019474, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:34.345690 18625 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:24:34.346354 18625 webserver.cc:460] Webserver started at http://127.2.32.62:43529/ using document root <none> and password file <none>
I20260621 05:24:34.346618 18625 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:34.346691 18625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:34.348517 18625 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:34.349629 18640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:34.350250 18625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:34.350481 18625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "cb4f712f75ca4b649cf74281f34999e1"
format_stamp: "Formatted at 2026-06-21 05:23:52 on dist-test-slave-b357"
server_key: "ae9ec513fcf2b6d2a85e412b02e71fa9"
server_key_iv: "2a08e8ce492227228a9f81f632e2df00"
server_key_version: "encryptionkey@0"
I20260621 05:24:34.351050 18625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:34.386211 18625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:34.389396 18625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:34.389668 18625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:34.395295 18625 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:40701
I20260621 05:24:34.395339 18692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:40701 every 8 connection(s)
I20260621 05:24:34.395779 18625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:24:34.396358 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 18625
I20260621 05:24:34.396669 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:36527
--local_ip_for_outbound_sockets=127.2.32.1
--tserver_master_addrs=127.2.32.62:40701
--webserver_port=38945
--webserver_interface=127.2.32.1
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:24:34.397593 18693 sys_catalog.cc:263] Verifying existing consensus state
I20260621 05:24:34.398252 18693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1: Bootstrap starting.
I20260621 05:24:34.411536 18693 log.cc:824] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:34.413836 18693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:24:34.414228 18693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1: Bootstrap complete.
I20260621 05:24:34.416543 18693 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f712f75ca4b649cf74281f34999e1" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40701 } }
I20260621 05:24:34.416791 18693 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb4f712f75ca4b649cf74281f34999e1, State: Initialized, Role: FOLLOWER
I20260621 05:24:34.416970 18693 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f712f75ca4b649cf74281f34999e1" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40701 } }
I20260621 05:24:34.417045 18693 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:24:34.417096 18693 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:24:34.417163 18693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 2 FOLLOWER]: Advancing to term 3
I20260621 05:24:34.418155 18693 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f712f75ca4b649cf74281f34999e1" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40701 } }
I20260621 05:24:34.418272 18693 leader_election.cc:302] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb4f712f75ca4b649cf74281f34999e1; no voters:
I20260621 05:24:34.418450 18693 leader_election.cc:288] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260621 05:24:34.418506 18697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 3 FOLLOWER]: Leader election won for term 3
I20260621 05:24:34.418650 18697 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [term 3 LEADER]: Becoming Leader. State: Replica: cb4f712f75ca4b649cf74281f34999e1, State: Running, Role: LEADER
I20260621 05:24:34.418715 18693 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:24:34.418735 18697 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f712f75ca4b649cf74281f34999e1" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40701 } }
I20260621 05:24:34.418990 18697 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb4f712f75ca4b649cf74281f34999e1. Latest consensus state: current_term: 3 leader_uuid: "cb4f712f75ca4b649cf74281f34999e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f712f75ca4b649cf74281f34999e1" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40701 } } }
I20260621 05:24:34.419047 18697 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [sys.catalog]: This master's current role is: LEADER
I20260621 05:24:34.419574 18698 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cb4f712f75ca4b649cf74281f34999e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f712f75ca4b649cf74281f34999e1" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 40701 } } }
I20260621 05:24:34.419654 18698 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1 [sys.catalog]: This master's current role is: LEADER
I20260621 05:24:34.420150 18706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:24:34.420948 18706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:24:34.421058 18706 catalog_manager.cc:1259] Loaded cluster ID: 0c656a092a7044f6b9b98ad26355ee4a
I20260621 05:24:34.421288 18706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:24:34.422443 18706 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:24:34.422613 18706 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1: Loaded TSK: 0
I20260621 05:24:34.422672 18706 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cb4f712f75ca4b649cf74281f34999e1: Loaded TSK: 1
F20260621 05:24:34.422884 18706 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd9ff2819ad google::LogMessage::Fail() at ??:0
@ 0x7fd9ff28488a google::LogMessage::SendToLog() at ??:0
@ 0x7fd9ff2814de google::LogMessage::Flush() at ??:0
@ 0x7fd9ff282a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fda019ee104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fda019ebd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fda071db1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fda071db48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fda071db594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fda071a1c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fda071abad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fda0114f0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fda01144954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fda01fa86db start_thread at ??:0
@ 0x7fd9fdd7f61f clone at ??:0
W20260621 05:24:34.532958 18695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:34.533249 18695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:34.533293 18695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:34.535521 18695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:34.535585 18695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:34.535627 18695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:24:34.537883 18695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:36527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=38945
--tserver_master_addrs=127.2.32.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:34.538163 18695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:34.538446 18695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:34.545859 18735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:34.546054 18737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:34.545997 18734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:34.546681 18695 server_base.cc:1034] running on GCE node
I20260621 05:24:34.547766 18695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:34.548187 18695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:34.549970 18695 hybrid_clock.cc:648] HybridClock initialized: now 1782019474549930 us; error 67 us; skew 500 ppm
Jun 21 05:24:34 dist-test-slave-b357 krb5kdc[17554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019474, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:34.553112 18695 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:24:34.553586 18695 webserver.cc:460] Webserver started at http://127.2.32.1:38945/ using document root <none> and password file <none>
I20260621 05:24:34.553834 18695 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:34.553906 18695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:34.555347 18695 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:34.556227 18744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:34.556445 18695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:34.556551 18695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "11ba9033670a40f3b1dcd0cb865fa607"
format_stamp: "Formatted at 2026-06-21 05:23:52 on dist-test-slave-b357"
server_key: "01c6bdf916ebb244e1e9bb0f1d90f54f"
server_key_iv: "f5380ecc90cbc280d3b1e531e7ddf343"
server_key_version: "encryptionkey@0"
I20260621 05:24:34.556929 18695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:34.577718 18695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:34.583534 18695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:34.586122 18695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:34.589767 18695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:34.590922 18751 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 05:24:34.593014 18695 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 05:24:34.593189 18695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260621 05:24:34.593238 18695 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 05:24:34.593891 18695 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 05:24:34.593950 18695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260621 05:24:34.594013 18751 tablet_bootstrap.cc:492] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607: Bootstrap starting.
I20260621 05:24:34.602797 18695 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:36527
I20260621 05:24:34.603291 18858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:36527 every 8 connection(s)
I20260621 05:24:34.603639 18695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:24:34.610073 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 18695
I20260621 05:24:34.610492 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:37849
--local_ip_for_outbound_sockets=127.2.32.2
--tserver_master_addrs=127.2.32.62:40701
--webserver_port=33649
--webserver_interface=127.2.32.2
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:24:34.612741 18751 log.cc:824] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:34.616072 18751 tablet_bootstrap.cc:492] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 05:24:34.616421 18751 tablet_bootstrap.cc:492] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607: Bootstrap complete.
I20260621 05:24:34.617774 18751 ts_tablet_manager.cc:1397] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.007s
I20260621 05:24:34.619712 18751 raft_consensus.cc:348] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } }
I20260621 05:24:34.620114 18751 raft_consensus.cc:729] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11ba9033670a40f3b1dcd0cb865fa607, State: Initialized, Role: FOLLOWER
I20260621 05:24:34.620277 18751 consensus_queue.cc:260] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } }
I20260621 05:24:34.620613 18751 ts_tablet_manager.cc:1428] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
W20260621 05:24:34.712957 18862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:34.713235 18862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:34.713300 18862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:34.714603 18862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:34.714643 18862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:34.714674 18862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
W20260621 05:24:34.715286 18861 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:24:34.604946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.1:59395)
0621 05:24:34.606104 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:34.606161 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:34.606886 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:34.715032 (+108146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1097,"thread_start_us":900,"threads_started":1}
W20260621 05:24:34.715548 18859 heartbeater.cc:644] Failed to heartbeat to 127.2.32.62:40701 (0 consecutive failures): Network error: Failed to ping master at 127.2.32.62:40701: Client connection negotiation failed: client connection to 127.2.32.62:40701: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 05:24:34.715863 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:34.715749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.1:49889)
0621 05:24:34.715804 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:34.715839 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260621 05:24:34.716228 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:34.716113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.1:51925)
0621 05:24:34.716190 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:34.716212 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260621 05:24:34.716540 18859 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 05:24:34.716670 18862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:37849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=33649
--tserver_master_addrs=127.2.32.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:34.716930 18862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:34.717180 18862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:34.722419 18869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:34.722448 18872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:34.722551 18862 server_base.cc:1034] running on GCE node
W20260621 05:24:34.722450 18870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:34.722882 18862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:34.723258 18862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:34.724473 18862 hybrid_clock.cc:648] HybridClock initialized: now 1782019474724444 us; error 41 us; skew 500 ppm
Jun 21 05:24:34 dist-test-slave-b357 krb5kdc[17554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019474, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:34.727999 18862 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:24:34.728675 18862 webserver.cc:460] Webserver started at http://127.2.32.2:33649/ using document root <none> and password file <none>
I20260621 05:24:34.728953 18862 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:34.729028 18862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:34.730654 18862 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:34.731542 18879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:34.731776 18862 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:34.731889 18862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "be235d3c549d44f0a9b4924326f225cb"
format_stamp: "Formatted at 2026-06-21 05:23:52 on dist-test-slave-b357"
server_key: "5483910a424db3844a921432c28d7377"
server_key_iv: "bafd5d9517524dc182486b91aa4831ee"
server_key_version: "encryptionkey@0"
I20260621 05:24:34.732300 18862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:34.757336 18862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:34.760371 18862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:34.760535 18862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:34.760834 18862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:34.761145 18862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:34.761199 18862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:34.761243 18862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:34.761269 18862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:34.768293 18862 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:37849
I20260621 05:24:34.768340 18992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:37849 every 8 connection(s)
I20260621 05:24:34.768711 18862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:24:34.769842 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:34.769403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.2:46315)
0621 05:24:34.769618 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:34.769690 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
W20260621 05:24:34.770417 18993 heartbeater.cc:644] Failed to heartbeat to 127.2.32.62:40701 (0 consecutive failures): Network error: Failed to ping master at 127.2.32.62:40701: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
I20260621 05:24:34.770777 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:34.770645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.2:36165)
0621 05:24:34.770721 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:34.770754 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260621 05:24:34.771191 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:34.771059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.2:42805)
0621 05:24:34.771146 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:34.771173 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260621 05:24:34.771335 18993 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 05:24:34.777555 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 18862
I20260621 05:24:34.777889 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:40135
--local_ip_for_outbound_sockets=127.2.32.3
--tserver_master_addrs=127.2.32.62:40701
--webserver_port=37085
--webserver_interface=127.2.32.3
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:24:34.864848 18996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:34.865222 18996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:34.865274 18996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:34.867735 18996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:34.867873 18996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:34.867950 18996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:24:34.870221 18996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:40135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=37085
--tserver_master_addrs=127.2.32.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:34.870604 18996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:34.870931 18996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:34.877032 19004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:34.877035 19001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:34.877070 19002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:34.877390 18996 server_base.cc:1034] running on GCE node
I20260621 05:24:34.877619 18996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:34.877871 18996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:34.879024 18996 hybrid_clock.cc:648] HybridClock initialized: now 1782019474878977 us; error 65 us; skew 500 ppm
Jun 21 05:24:34 dist-test-slave-b357 krb5kdc[17554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019474, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:34.881798 18996 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:24:34.882316 18996 webserver.cc:460] Webserver started at http://127.2.32.3:37085/ using document root <none> and password file <none>
I20260621 05:24:34.882546 18996 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:34.882613 18996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:34.884423 18996 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:24:34.885484 19011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:34.885795 18996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:34.886098 18996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "d7bb06947ddb440fafc0bf38fa8b1037"
format_stamp: "Formatted at 2026-06-21 05:23:53 on dist-test-slave-b357"
server_key: "b4bce4edab6920e49a556c9516153204"
server_key_iv: "76cad499582e59b19331684fa617939a"
server_key_version: "encryptionkey@0"
I20260621 05:24:34.886683 18996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:34.906981 18996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:34.911151 18996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:34.911352 18996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:34.911628 18996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:34.911959 18996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:34.912014 18996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:34.912055 18996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:34.912117 18996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:34.918133 18996 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:40135
I20260621 05:24:34.918175 19124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:40135 every 8 connection(s)
I20260621 05:24:34.918563 18996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:24:34.919812 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:34.919324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.3:53023)
0621 05:24:34.919606 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:34.919674 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":81,"threads_started":1}
W20260621 05:24:34.920204 19125 heartbeater.cc:644] Failed to heartbeat to 127.2.32.62:40701 (0 consecutive failures): Network error: Failed to ping master at 127.2.32.62:40701: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
I20260621 05:24:34.920488 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:34.920391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.3:41683)
0621 05:24:34.920441 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:34.920472 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260621 05:24:34.920763 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:34.920677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.3:42133)
0621 05:24:34.920719 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:34.920741 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260621 05:24:34.920861 19125 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 05:24:34.924321 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 18996
I20260621 05:24:35.717190 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:35.716916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.1:46119)
0621 05:24:35.717074 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:35.717120 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
I20260621 05:24:35.772009 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:35.771727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.2:37737)
0621 05:24:35.771911 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:35.771953 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":75,"threads_started":1}
I20260621 05:24:35.921612 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:35.921329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.3:51639)
0621 05:24:35.921521 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:35.921563 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":84,"threads_started":1}
I20260621 05:24:36.299018 19132 raft_consensus.cc:482] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:24:36.299214 19132 raft_consensus.cc:504] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } }
I20260621 05:24:36.299629 19132 leader_election.cc:288] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7bb06947ddb440fafc0bf38fa8b1037 (127.2.32.3:40135), be235d3c549d44f0a9b4924326f225cb (127.2.32.2:37849)
Jun 21 05:24:36 dist-test-slave-b357 krb5kdc[17554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019474, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.3@KRBTEST.COM
Jun 21 05:24:36 dist-test-slave-b357 krb5kdc[17554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019474, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.2@KRBTEST.COM
I20260621 05:24:36.314312 19135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:36.300024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:37849 (local address 127.2.32.1:48157)
0621 05:24:36.300167 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:36.300176 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:36.300221 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:36.301183 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:36.301184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:36.301597 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:36.301886 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:36.301894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:36.303955 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:36.303957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:36.305529 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:36.305533 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:36.306496 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:36.306496 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:36.306667 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:36.307683 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:24:36.307690 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:36.310157 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:36.312738 (+ 2581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:36.312740 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:36.312740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:36.312982 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:36.313233 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:36.313234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:36.313234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:36.313353 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:36.313751 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:36.313757 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:36.313891 (+ 134us) client_negotiation.cc:769] Sending connection context
0621 05:24:36.314006 (+ 115us) client_negotiation.cc:240] Negotiation successful
0621 05:24:36.314210 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":92,"thread_start_us":41,"threads_started":1}
I20260621 05:24:36.314327 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:36.299959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:36769 (local address 127.2.32.3:40135)
0621 05:24:36.300176 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:36.300190 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:36.300225 (+ 35us) server_negotiation.cc:407] Connection header received
0621 05:24:36.300817 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:36.300820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:36.300866 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:36.301042 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:36.302113 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:36.303766 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:36.305686 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:36.306349 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:36.306477 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:36.309959 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:36.309966 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:36.309967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:36.309985 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:36.312435 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:36.312990 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:36.312995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:36.312995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:36.313150 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:36.313418 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:36.313420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:36.313420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:36.313676 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:36.313775 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:36.314065 (+ 290us) server_negotiation.cc:299] Negotiation successful
0621 05:24:36.314208 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":74,"threads_started":1}
I20260621 05:24:36.314327 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:36.300158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:48157 (local address 127.2.32.2:37849)
0621 05:24:36.300304 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:36.300313 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:36.300329 (+ 16us) server_negotiation.cc:407] Connection header received
0621 05:24:36.300834 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:36.300837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:36.300886 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:36.301083 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:36.302113 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:36.303768 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:36.305692 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:36.306349 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:36.306477 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:36.310282 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:36.310286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:36.310286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:36.310296 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:36.312598 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:36.313076 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:36.313078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:36.313079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:36.313156 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:36.313442 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:36.313443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:36.313443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:36.313674 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:36.313772 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:36.314056 (+ 284us) server_negotiation.cc:299] Negotiation successful
0621 05:24:36.314193 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":55,"threads_started":1}
I20260621 05:24:36.314420 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:36.299819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:40135 (local address 127.2.32.1:36769)
0621 05:24:36.299991 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:36.300003 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:36.300103 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:36.301121 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:36.301124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:36.301597 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:36.301886 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:36.301894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:36.303967 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:36.303967 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:36.305530 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:36.305533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:36.306495 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:36.306496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:36.306668 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:36.307678 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:24:36.307690 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:36.309719 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:36.312538 (+ 2819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:36.312546 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:36.312549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:36.312872 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:36.313231 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:36.313232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:36.313233 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:36.313348 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:36.313753 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:36.313758 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:36.313891 (+ 133us) client_negotiation.cc:769] Sending connection context
0621 05:24:36.314008 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 05:24:36.314290 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":269,"thread_start_us":88,"threads_started":1}
W20260621 05:24:36.315789 18748 leader_election.cc:341] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be235d3c549d44f0a9b4924326f225cb (127.2.32.2:37849): Not found: Tablet not found: 88535e932a6045c396770c09981bb6fe
W20260621 05:24:36.315881 18748 leader_election.cc:341] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7bb06947ddb440fafc0bf38fa8b1037 (127.2.32.3:40135): Not found: Tablet not found: 88535e932a6045c396770c09981bb6fe
I20260621 05:24:36.315900 18748 leader_election.cc:302] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11ba9033670a40f3b1dcd0cb865fa607; no voters: be235d3c549d44f0a9b4924326f225cb, d7bb06947ddb440fafc0bf38fa8b1037
I20260621 05:24:36.316079 19132 raft_consensus.cc:2726] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:24:36.718080 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:36.717887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.1:36299)
0621 05:24:36.717966 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:36.718021 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260621 05:24:36.773025 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:36.772600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.2:35515)
0621 05:24:36.772923 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:36.772970 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":234,"threads_started":1}
I20260621 05:24:36.922685 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:36.922396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.3:54997)
0621 05:24:36.922586 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:36.922643 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":79,"threads_started":1}
I20260621 05:24:37.718938 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:37.718673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.1:56439)
0621 05:24:37.718840 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:37.718888 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":77,"threads_started":1}
I20260621 05:24:37.773910 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:37.773681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.2:45591)
0621 05:24:37.773820 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:37.773858 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
I20260621 05:24:37.923825 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:37.923514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.3:35433)
0621 05:24:37.923705 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:37.923762 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":97,"threads_started":1}
I20260621 05:24:38.014846 19142 raft_consensus.cc:482] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:24:38.015020 19142 raft_consensus.cc:504] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } }
I20260621 05:24:38.015319 19142 leader_election.cc:288] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7bb06947ddb440fafc0bf38fa8b1037 (127.2.32.3:40135), be235d3c549d44f0a9b4924326f225cb (127.2.32.2:37849)
W20260621 05:24:38.015691 18748 leader_election.cc:341] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7bb06947ddb440fafc0bf38fa8b1037 (127.2.32.3:40135): Not found: Tablet not found: 88535e932a6045c396770c09981bb6fe
W20260621 05:24:38.015789 18748 leader_election.cc:341] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be235d3c549d44f0a9b4924326f225cb (127.2.32.2:37849): Not found: Tablet not found: 88535e932a6045c396770c09981bb6fe
I20260621 05:24:38.015867 18748 leader_election.cc:302] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11ba9033670a40f3b1dcd0cb865fa607; no voters: be235d3c549d44f0a9b4924326f225cb, d7bb06947ddb440fafc0bf38fa8b1037
I20260621 05:24:38.016135 19142 raft_consensus.cc:2726] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:24:38.720000 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:38.719620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.1:54397)
0621 05:24:38.719849 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:38.719930 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":70,"threads_started":1}
I20260621 05:24:38.774670 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:38.774424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.2:51415)
0621 05:24:38.774584 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:38.774627 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":63,"threads_started":1}
I20260621 05:24:38.924916 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:38.924500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.3:43873)
0621 05:24:38.924676 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:38.924843 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":81,"threads_started":1}
I20260621 05:24:39.720897 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:39.720656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.1:41909)
0621 05:24:39.720808 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:39.720847 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":64,"threads_started":1}
I20260621 05:24:39.775504 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:39.775295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.2:47123)
0621 05:24:39.775416 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:39.775459 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260621 05:24:39.925879 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:39.925533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.3:35215)
0621 05:24:39.925765 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:39.925821 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":106,"threads_started":1}
I20260621 05:24:40.722114 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:40.721683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.1:48181)
0621 05:24:40.721979 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:40.722049 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":111,"threads_started":1}
I20260621 05:24:40.776400 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:40.776103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.2:34525)
0621 05:24:40.776312 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:40.776352 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":162,"threads_started":1}
I20260621 05:24:40.926800 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:40.926549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.3:39919)
0621 05:24:40.926717 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:40.926755 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260621 05:24:41.281204 19152 raft_consensus.cc:482] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:24:41.281308 19152 raft_consensus.cc:504] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } }
I20260621 05:24:41.281488 19152 leader_election.cc:288] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7bb06947ddb440fafc0bf38fa8b1037 (127.2.32.3:40135), be235d3c549d44f0a9b4924326f225cb (127.2.32.2:37849)
W20260621 05:24:41.282250 18748 leader_election.cc:341] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7bb06947ddb440fafc0bf38fa8b1037 (127.2.32.3:40135): Not found: Tablet not found: 88535e932a6045c396770c09981bb6fe
W20260621 05:24:41.282346 18748 leader_election.cc:341] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be235d3c549d44f0a9b4924326f225cb (127.2.32.2:37849): Not found: Tablet not found: 88535e932a6045c396770c09981bb6fe
I20260621 05:24:41.282363 18748 leader_election.cc:302] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11ba9033670a40f3b1dcd0cb865fa607; no voters: be235d3c549d44f0a9b4924326f225cb, d7bb06947ddb440fafc0bf38fa8b1037
I20260621 05:24:41.282433 19152 raft_consensus.cc:2726] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:24:41.722891 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:41.722653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.1:50961)
0621 05:24:41.722804 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:41.722842 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260621 05:24:41.777223 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:41.776986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.2:53493)
0621 05:24:41.777135 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:41.777179 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260621 05:24:41.927786 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:41.927477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.3:46123)
0621 05:24:41.927629 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:41.927724 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260621 05:24:42.724273 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:42.723930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.1:44801)
0621 05:24:42.724163 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:42.724213 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":115,"threads_started":1}
I20260621 05:24:42.777984 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:42.777749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.2:50783)
0621 05:24:42.777898 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:42.777943 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":62,"threads_started":1}
I20260621 05:24:42.928755 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:42.928455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.3:59381)
0621 05:24:42.928669 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:42.928715 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
I20260621 05:24:43.725291 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:43.724975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.1:40303)
0621 05:24:43.725170 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:43.725242 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":94,"threads_started":1}
I20260621 05:24:43.778733 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:43.778463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.2:47585)
0621 05:24:43.778616 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:43.778659 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260621 05:24:43.929656 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:43.929327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.3:54685)
0621 05:24:43.929526 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:43.929582 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
I20260621 05:24:44.726528 19162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:44.726132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.1:59739)
0621 05:24:44.726353 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:44.726437 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":102,"threads_started":1}
I20260621 05:24:44.752535 19163 raft_consensus.cc:482] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:24:44.752652 19163 raft_consensus.cc:504] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bb06947ddb440fafc0bf38fa8b1037" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 40135 } } peers { permanent_uuid: "11ba9033670a40f3b1dcd0cb865fa607" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36527 } } peers { permanent_uuid: "be235d3c549d44f0a9b4924326f225cb" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 37849 } }
I20260621 05:24:44.752833 19163 leader_election.cc:288] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7bb06947ddb440fafc0bf38fa8b1037 (127.2.32.3:40135), be235d3c549d44f0a9b4924326f225cb (127.2.32.2:37849)
W20260621 05:24:44.753203 18748 leader_election.cc:341] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be235d3c549d44f0a9b4924326f225cb (127.2.32.2:37849): Not found: Tablet not found: 88535e932a6045c396770c09981bb6fe
W20260621 05:24:44.753274 18748 leader_election.cc:341] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7bb06947ddb440fafc0bf38fa8b1037 (127.2.32.3:40135): Not found: Tablet not found: 88535e932a6045c396770c09981bb6fe
I20260621 05:24:44.753291 18748 leader_election.cc:302] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11ba9033670a40f3b1dcd0cb865fa607; no voters: be235d3c549d44f0a9b4924326f225cb, d7bb06947ddb440fafc0bf38fa8b1037
I20260621 05:24:44.753409 19163 raft_consensus.cc:2726] T 88535e932a6045c396770c09981bb6fe P 11ba9033670a40f3b1dcd0cb865fa607 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 05:24:44.779616 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:44.779340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.2:58397)
0621 05:24:44.779522 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:44.779570 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":69,"threads_started":1}
I20260621 05:24:44.930718 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:44.930372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.3:33561)
0621 05:24:44.930631 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:44.930671 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":167,"threads_started":1}
I20260621 05:24:45.727826 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:45.727462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.1:40013)
0621 05:24:45.727647 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:45.727760 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
I20260621 05:24:45.780524 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:45.780260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.2:56407)
0621 05:24:45.780422 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:45.780475 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":65,"threads_started":1}
I20260621 05:24:45.931741 19168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:45.931484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.3:57861)
0621 05:24:45.931659 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:45.931695 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
I20260621 05:24:46.728894 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:46.728498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.1:47031)
0621 05:24:46.728773 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:46.728818 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":127,"threads_started":1}
I20260621 05:24:46.781414 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:46.781101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.2:38463)
0621 05:24:46.781288 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:46.781349 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":64,"threads_started":1}
I20260621 05:24:46.932658 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:46.932368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.3:47171)
0621 05:24:46.932550 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:46.932610 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":89,"threads_started":1}
I20260621 05:24:47.729907 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:47.729546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.1:37055)
0621 05:24:47.729790 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:47.729840 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":92,"threads_started":1}
I20260621 05:24:47.782558 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:47.782164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.2:37615)
0621 05:24:47.782437 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:47.782492 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":155,"threads_started":1}
I20260621 05:24:47.933571 19174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:47.933279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.3:45617)
0621 05:24:47.933482 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:47.933520 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
I20260621 05:24:48.730968 19175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:48.730611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.1:60435)
0621 05:24:48.730835 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:48.730902 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":107,"threads_started":1}
I20260621 05:24:48.783488 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:48.783272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.2:53363)
0621 05:24:48.783400 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:48.783438 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260621 05:24:48.934291 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:48.934068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.3:44563)
0621 05:24:48.934213 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:48.934247 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260621 05:24:49.731874 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:49.731633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.1:52453)
0621 05:24:49.731789 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:49.731828 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":63,"threads_started":1}
I20260621 05:24:49.784499 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:49.784187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.2:49665)
0621 05:24:49.784398 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:49.784451 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":113,"threads_started":1}
I20260621 05:24:49.926282 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 18695
I20260621 05:24:49.931421 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 18862
I20260621 05:24:49.935103 19180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:49.934880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:40701 (local address 127.2.32.3:56357)
0621 05:24:49.935004 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:49.935045 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.62:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":52,"threads_started":1}
I20260621 05:24:49.936386 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 18996
2026-06-21T05:24:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58598 ms)
[----------] 30 tests from SecurityITest (264831 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:24:50 dist-test-slave-b357 krb5kdc[19183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:24:50 dist-test-slave-b357 krb5kdc[19183](info): set up 2 sockets
Jun 21 05:24:50 dist-test-slave-b357 krb5kdc[19183](info): commencing operation
krb5kdc: starting...
W20260621 05:24:50.999675 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:24:51 dist-test-slave-b357 krb5kdc[19183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:24:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:24:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:24:51.125604 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:36785
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:36785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:24:51.205025 19196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:51.205216 19196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:51.205237 19196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:51.206357 19196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:24:51.206399 19196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:51.206413 19196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:51.206426 19196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:24:51.206439 19196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:24:51.207937 19196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:36785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:36785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:51.208163 19196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:51.208379 19196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:51.213577 19202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:51.213575 19201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:51.213619 19204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:51.213581 19196 server_base.cc:1034] running on GCE node
I20260621 05:24:51.213986 19196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:51.214212 19196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:51.215399 19196 hybrid_clock.cc:648] HybridClock initialized: now 1782019491215389 us; error 34 us; skew 500 ppm
Jun 21 05:24:51 dist-test-slave-b357 krb5kdc[19183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019491, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:51.217898 19196 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:24:51.218364 19196 webserver.cc:460] Webserver started at http://127.2.32.62:38199/ using document root <none> and password file <none>
I20260621 05:24:51.218626 19196 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:51.218672 19196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:51.218760 19196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:51.219808 19196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "236abdc5c4b046bead7e80c2c99e36de"
format_stamp: "Formatted at 2026-06-21 05:24:51 on dist-test-slave-b357"
server_key: "ce7e52edc0a8e9c5d6b68f24d0130953"
server_key_iv: "68ca9e93c6ef34f04dac20fe4a0e9d09"
server_key_version: "encryptionkey@0"
I20260621 05:24:51.220114 19196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "236abdc5c4b046bead7e80c2c99e36de"
format_stamp: "Formatted at 2026-06-21 05:24:51 on dist-test-slave-b357"
server_key: "ce7e52edc0a8e9c5d6b68f24d0130953"
server_key_iv: "68ca9e93c6ef34f04dac20fe4a0e9d09"
server_key_version: "encryptionkey@0"
I20260621 05:24:51.221383 19196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:24:51.222177 19211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:51.222406 19196 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:24:51.222528 19196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "236abdc5c4b046bead7e80c2c99e36de"
format_stamp: "Formatted at 2026-06-21 05:24:51 on dist-test-slave-b357"
server_key: "ce7e52edc0a8e9c5d6b68f24d0130953"
server_key_iv: "68ca9e93c6ef34f04dac20fe4a0e9d09"
server_key_version: "encryptionkey@0"
I20260621 05:24:51.222602 19196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:51.230859 19196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:51.233274 19196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:51.233449 19196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:51.237520 19196 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:36785
I20260621 05:24:51.237555 19263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:36785 every 8 connection(s)
I20260621 05:24:51.237886 19196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:24:51.238801 19264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:51.240289 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 19196
I20260621 05:24:51.240391 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:24:51.240612 2176 external_mini_cluster.cc:1428] Setting key e45478c7ea82c3effc9ca50efa392379
I20260621 05:24:51.241285 19264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de: Bootstrap starting.
I20260621 05:24:51.241972 19264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:51.242296 19264 log.cc:824] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:51.242938 19264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de: No bootstrap required, opened a new log
I20260621 05:24:51.245169 19264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236abdc5c4b046bead7e80c2c99e36de" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36785 } }
I20260621 05:24:51.245328 19264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:51.245358 19264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 236abdc5c4b046bead7e80c2c99e36de, State: Initialized, Role: FOLLOWER
I20260621 05:24:51.245478 19264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236abdc5c4b046bead7e80c2c99e36de" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36785 } }
I20260621 05:24:51.245550 19264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:24:51.245579 19264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:24:51.245622 19264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:51.246213 19264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236abdc5c4b046bead7e80c2c99e36de" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36785 } }
I20260621 05:24:51.246326 19264 leader_election.cc:302] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 236abdc5c4b046bead7e80c2c99e36de; no voters:
I20260621 05:24:51.246500 19264 leader_election.cc:288] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:24:51.246516 19269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:24:51.246701 19269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de [term 1 LEADER]: Becoming Leader. State: Replica: 236abdc5c4b046bead7e80c2c99e36de, State: Running, Role: LEADER
I20260621 05:24:51.246809 19264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:24:51.246845 19269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236abdc5c4b046bead7e80c2c99e36de" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36785 } }
I20260621 05:24:51.247268 19269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 236abdc5c4b046bead7e80c2c99e36de. Latest consensus state: current_term: 1 leader_uuid: "236abdc5c4b046bead7e80c2c99e36de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236abdc5c4b046bead7e80c2c99e36de" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36785 } } }
I20260621 05:24:51.247279 19270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "236abdc5c4b046bead7e80c2c99e36de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236abdc5c4b046bead7e80c2c99e36de" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 36785 } } }
I20260621 05:24:51.247574 19269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de [sys.catalog]: This master's current role is: LEADER
I20260621 05:24:51.247587 19270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de [sys.catalog]: This master's current role is: LEADER
I20260621 05:24:51.247828 19278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 21 05:24:51 dist-test-slave-b357 krb5kdc[19183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:24:51.248306 19278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:24:51.250340 19278 catalog_manager.cc:1347] Generated new cluster ID: 0408e3b4957940d38e779387eb083153
I20260621 05:24:51.250396 19278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:24:51.252206 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.241126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.2.32.62:36785)
0621 05:24:51.241466 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:51.241476 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:51.241508 (+ 32us) server_negotiation.cc:407] Connection header received
0621 05:24:51.242146 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:51.242156 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:51.242217 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:51.242380 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:51.243034 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.244415 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.245162 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.245751 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.245858 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:51.248752 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:51.248755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:51.248756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:51.248767 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:51.250701 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:51.251080 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:51.251083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:51.251083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:51.251198 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:51.251407 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:51.251407 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:51.251408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:51.251629 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:51.251804 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:51.251866 (+ 62us) server_negotiation.cc:299] Negotiation successful
0621 05:24:51.252001 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":201,"threads_started":1}
I20260621 05:24:51.271942 19278 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:24:51.272624 19278 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:24:51.276561 19278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de: Generated new TSK 0
I20260621 05:24:51.276799 19278 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
I20260621 05:24:51.306293 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:36785
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:24:51.394277 19290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:51.394507 19290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:51.394543 19290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:51.395669 19290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:51.395735 19290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:51.395783 19290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:24:51.397214 19290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:36785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:51.397471 19290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:51.397691 19290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:51.402871 19298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:51.402890 19295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:51.402840 19296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:51.403193 19290 server_base.cc:1034] running on GCE node
I20260621 05:24:51.403556 19290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:51.403870 19290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:51.405093 19290 hybrid_clock.cc:648] HybridClock initialized: now 1782019491405064 us; error 50 us; skew 500 ppm
Jun 21 05:24:51 dist-test-slave-b357 krb5kdc[19183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019491, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:51.408103 19290 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:24:51.408686 19290 webserver.cc:460] Webserver started at http://127.2.32.1:42285/ using document root <none> and password file <none>
I20260621 05:24:51.408958 19290 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:51.409030 19290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:51.409186 19290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:51.410337 19290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "c6f05422b1d441909a1c6ccd75d38757"
format_stamp: "Formatted at 2026-06-21 05:24:51 on dist-test-slave-b357"
server_key: "8b51422b009788f2c5c1c1c85219f98b"
server_key_iv: "eacd5fa2d6c250a8a02575888f699d1e"
server_key_version: "encryptionkey@0"
I20260621 05:24:51.410723 19290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "c6f05422b1d441909a1c6ccd75d38757"
format_stamp: "Formatted at 2026-06-21 05:24:51 on dist-test-slave-b357"
server_key: "8b51422b009788f2c5c1c1c85219f98b"
server_key_iv: "eacd5fa2d6c250a8a02575888f699d1e"
server_key_version: "encryptionkey@0"
I20260621 05:24:51.412307 19290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:24:51.413555 19305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:51.413834 19290 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:51.413969 19290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "c6f05422b1d441909a1c6ccd75d38757"
format_stamp: "Formatted at 2026-06-21 05:24:51 on dist-test-slave-b357"
server_key: "8b51422b009788f2c5c1c1c85219f98b"
server_key_iv: "eacd5fa2d6c250a8a02575888f699d1e"
server_key_version: "encryptionkey@0"
I20260621 05:24:51.414073 19290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:51.436249 19290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:51.438548 19290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:51.438728 19290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:51.438988 19290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:51.439376 19290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:51.439428 19290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:51.439471 19290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:51.439499 19290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:51.445565 19290 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:41629
I20260621 05:24:51.445608 19418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:41629 every 8 connection(s)
I20260621 05:24:51.445921 19290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:24:51 dist-test-slave-b357 krb5kdc[19183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019491, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:24:51.453930 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 19290
I20260621 05:24:51.454062 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:24:51.454293 2176 external_mini_cluster.cc:1428] Setting key a17b68012abda2d8efebebe27833d3a1
I20260621 05:24:51.456019 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.446678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:36391 (local address 127.2.32.62:36785)
0621 05:24:51.446750 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:51.446753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:51.446961 (+ 208us) server_negotiation.cc:407] Connection header received
0621 05:24:51.447565 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:51.447566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:51.447577 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:51.447628 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:51.448781 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.449384 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.450475 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.450733 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.450823 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:51.453084 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:51.453086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:51.453087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:51.453100 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:51.454641 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:51.455059 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:51.455060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:51.455061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:51.455117 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:51.455375 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:51.455376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:51.455376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:51.455498 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:51.455565 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:51.455835 (+ 270us) server_negotiation.cc:299] Negotiation successful
0621 05:24:51.455930 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 05:24:51.456291 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.446635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:36785 (local address 127.2.32.1:36391)
0621 05:24:51.446803 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:51.446839 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:51.447465 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:51.447693 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:51.447696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:51.448356 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:51.448607 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:51.448612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.449523 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.449524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:51.450363 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:51.450365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.450838 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.450839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:51.450944 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:51.451522 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:24:51.451529 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:51.452977 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:51.454711 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:51.454718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:51.454720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:51.454998 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:51.455209 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:51.455210 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:51.455210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:51.455307 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:51.455565 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:51.455572 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:51.455705 (+ 133us) client_negotiation.cc:769] Sending connection context
0621 05:24:51.455800 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 05:24:51.455958 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
I20260621 05:24:51.456785 19419 heartbeater.cc:347] Connected to a master server at 127.2.32.62:36785
I20260621 05:24:51.456889 19419 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:51.457079 19419 heartbeater.cc:511] Master 127.2.32.62:36785 requested a full tablet report, sending...
I20260621 05:24:51.457621 19228 ts_manager.cc:194] Registered new tserver with Master: c6f05422b1d441909a1c6ccd75d38757 (127.2.32.1:41629)
I20260621 05:24:51.458235 19228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:36391
WARNING: no policy specified for kudu/127.2.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
I20260621 05:24:51.482897 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:36785
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:24:51.567293 19424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:51.567495 19424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:51.567519 19424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:51.568668 19424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:51.568720 19424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:51.568746 19424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:24:51.570138 19424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:36785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:51.570358 19424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:51.570623 19424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:51.575944 19429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:51.575944 19430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:51.575974 19424 server_base.cc:1034] running on GCE node
W20260621 05:24:51.575978 19432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:51.576354 19424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:51.576601 19424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:51.577805 19424 hybrid_clock.cc:648] HybridClock initialized: now 1782019491577795 us; error 38 us; skew 500 ppm
Jun 21 05:24:51 dist-test-slave-b357 krb5kdc[19183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019491, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:51.580406 19424 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:24:51.580780 19424 webserver.cc:460] Webserver started at http://127.2.32.2:36769/ using document root <none> and password file <none>
I20260621 05:24:51.580993 19424 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:51.581048 19424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:51.581172 19424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:51.582206 19424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "56caa4f7346a4d43bceb10693f0e73bc"
format_stamp: "Formatted at 2026-06-21 05:24:51 on dist-test-slave-b357"
server_key: "b452fbfde0e53d295846e5790d4e1c99"
server_key_iv: "b908a07f55d50a41b1c87c6e90834e9a"
server_key_version: "encryptionkey@0"
I20260621 05:24:51.582546 19424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "56caa4f7346a4d43bceb10693f0e73bc"
format_stamp: "Formatted at 2026-06-21 05:24:51 on dist-test-slave-b357"
server_key: "b452fbfde0e53d295846e5790d4e1c99"
server_key_iv: "b908a07f55d50a41b1c87c6e90834e9a"
server_key_version: "encryptionkey@0"
I20260621 05:24:51.583887 19424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:24:51.584630 19439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:51.584798 19424 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260621 05:24:51.584892 19424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "56caa4f7346a4d43bceb10693f0e73bc"
format_stamp: "Formatted at 2026-06-21 05:24:51 on dist-test-slave-b357"
server_key: "b452fbfde0e53d295846e5790d4e1c99"
server_key_iv: "b908a07f55d50a41b1c87c6e90834e9a"
server_key_version: "encryptionkey@0"
I20260621 05:24:51.584965 19424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:51.601156 19424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:51.603698 19424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:51.603856 19424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:51.604080 19424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:51.604364 19424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:51.604413 19424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:51.604621 19424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:51.604707 19424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:51.612093 19424 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:45233
I20260621 05:24:51.612236 19552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:45233 every 8 connection(s)
I20260621 05:24:51.612557 19424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:24:51.618662 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 19424
I20260621 05:24:51.618798 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:24:51.618999 2176 external_mini_cluster.cc:1428] Setting key 9e78d1d7cacf1703726ccf53276436b3
Jun 21 05:24:51 dist-test-slave-b357 krb5kdc[19183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019491, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:24:51.622822 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.613609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:36785 (local address 127.2.32.2:58593)
0621 05:24:51.613840 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:51.613888 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:51.614401 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:51.614653 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:51.614656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:51.615316 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:51.615519 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:51.615525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.616229 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.616230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:51.617035 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:51.617038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.617423 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.617424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:51.617575 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:51.618097 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:24:51.618105 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:51.619765 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:51.621561 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:51.621566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:51.621568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:51.621775 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:51.621974 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:51.621975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:51.621975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:51.622068 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:51.622348 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:51.622352 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:51.622459 (+ 107us) client_negotiation.cc:769] Sending connection context
0621 05:24:51.622552 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 05:24:51.622665 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":119,"threads_started":1}
I20260621 05:24:51.622821 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.613659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:58593 (local address 127.2.32.62:36785)
0621 05:24:51.613777 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:51.613779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:51.613973 (+ 194us) server_negotiation.cc:407] Connection header received
0621 05:24:51.614509 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:51.614510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:51.614523 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:51.614579 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:51.615611 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.616129 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.617129 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.617328 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.617415 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:51.619860 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:51.619861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:51.619861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:51.619868 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:51.621475 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:51.621848 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:51.621850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:51.621850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:51.621910 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:51.622138 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:51.622139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:51.622139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:51.622282 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:51.622352 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:51.622595 (+ 243us) server_negotiation.cc:299] Negotiation successful
0621 05:24:51.622681 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 05:24:51.623257 19553 heartbeater.cc:347] Connected to a master server at 127.2.32.62:36785
I20260621 05:24:51.623363 19553 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:51.623562 19553 heartbeater.cc:511] Master 127.2.32.62:36785 requested a full tablet report, sending...
I20260621 05:24:51.623908 19228 ts_manager.cc:194] Registered new tserver with Master: 56caa4f7346a4d43bceb10693f0e73bc (127.2.32.2:45233)
I20260621 05:24:51.624243 19228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:58593
WARNING: no policy specified for kudu/127.2.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
I20260621 05:24:51.645961 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:36785
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:24:51.731654 19558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:51.731913 19558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:51.731953 19558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:51.733160 19558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:51.733227 19558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:51.733276 19558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:24:51.734874 19558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:36785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:51.735189 19558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:51.735422 19558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:51.740473 19563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:51.740515 19558 server_base.cc:1034] running on GCE node
W20260621 05:24:51.740485 19564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:51.740478 19566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:51.740864 19558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:51.741111 19558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:51.742268 19558 hybrid_clock.cc:648] HybridClock initialized: now 1782019491742264 us; error 37 us; skew 500 ppm
Jun 21 05:24:51 dist-test-slave-b357 krb5kdc[19183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019491, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:51.745297 19558 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:24:51.745730 19558 webserver.cc:460] Webserver started at http://127.2.32.3:34191/ using document root <none> and password file <none>
I20260621 05:24:51.745950 19558 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:51.746016 19558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:51.746163 19558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:51.747323 19558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "d004b03820ee4f62b46d7a993f326f97"
format_stamp: "Formatted at 2026-06-21 05:24:51 on dist-test-slave-b357"
server_key: "5787df5f7d9d5736272ab71a6c7e4677"
server_key_iv: "a3529a429bde6787edfa194b059c9d28"
server_key_version: "encryptionkey@0"
I20260621 05:24:51.747677 19558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "d004b03820ee4f62b46d7a993f326f97"
format_stamp: "Formatted at 2026-06-21 05:24:51 on dist-test-slave-b357"
server_key: "5787df5f7d9d5736272ab71a6c7e4677"
server_key_iv: "a3529a429bde6787edfa194b059c9d28"
server_key_version: "encryptionkey@0"
I20260621 05:24:51.749274 19558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:24:51.750145 19573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:51.750377 19558 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:51.750487 19558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "d004b03820ee4f62b46d7a993f326f97"
format_stamp: "Formatted at 2026-06-21 05:24:51 on dist-test-slave-b357"
server_key: "5787df5f7d9d5736272ab71a6c7e4677"
server_key_iv: "a3529a429bde6787edfa194b059c9d28"
server_key_version: "encryptionkey@0"
I20260621 05:24:51.750576 19558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:51.765280 19558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:51.767834 19558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:51.768002 19558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:51.768236 19558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:51.768596 19558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:51.768658 19558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:51.768697 19558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:51.768724 19558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:51.775296 19558 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:33405
I20260621 05:24:51.775353 19686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:33405 every 8 connection(s)
I20260621 05:24:51.775636 19558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
Jun 21 05:24:51 dist-test-slave-b357 krb5kdc[19183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019491, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:24:51.782670 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 19558
I20260621 05:24:51.782753 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:24:51.782954 2176 external_mini_cluster.cc:1428] Setting key 7dadf57557b77d1c0d009d3046546c5d
I20260621 05:24:51.785979 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.776367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:43347 (local address 127.2.32.62:36785)
0621 05:24:51.776427 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:51.776429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:51.776638 (+ 209us) server_negotiation.cc:407] Connection header received
0621 05:24:51.777182 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:51.777183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:51.777196 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:51.777247 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:51.778157 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.778656 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.779680 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.779883 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.779965 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:51.782393 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:51.782394 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:51.782394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:51.782401 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:51.784119 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:51.784604 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:51.784606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:51.784606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:51.784715 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:51.785207 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:51.785209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:51.785209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:51.785355 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:51.785433 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:51.785782 (+ 349us) server_negotiation.cc:299] Negotiation successful
0621 05:24:51.785895 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:24:51.786053 19689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.776355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:36785 (local address 127.2.32.3:43347)
0621 05:24:51.776503 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:51.776552 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:51.777074 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:51.777316 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:51.777319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:51.777876 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:51.778056 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:51.778059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.778744 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.778745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:51.779596 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:51.779598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.779975 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.779976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:51.780092 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:51.780676 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:24:51.780684 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:51.782296 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:51.784218 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:51.784224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:51.784226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:51.784464 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:51.784881 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:51.784883 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:51.784884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:51.785088 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:51.785467 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:51.785474 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:51.785601 (+ 127us) client_negotiation.cc:769] Sending connection context
0621 05:24:51.785718 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 05:24:51.785861 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260621 05:24:51.786624 19687 heartbeater.cc:347] Connected to a master server at 127.2.32.62:36785
I20260621 05:24:51.786746 19687 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:51.787081 19687 heartbeater.cc:511] Master 127.2.32.62:36785 requested a full tablet report, sending...
I20260621 05:24:51.787576 19228 ts_manager.cc:194] Registered new tserver with Master: d004b03820ee4f62b46d7a993f326f97 (127.2.32.3:33405)
I20260621 05:24:51.788084 19228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:43347
I20260621 05:24:51.794883 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:24:51.803364 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.796616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.2.32.62:36785)
0621 05:24:51.796692 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:51.796693 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:51.796752 (+ 59us) server_negotiation.cc:407] Connection header received
0621 05:24:51.796795 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:51.796796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:51.796805 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:51.796862 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:51.797434 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.798016 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.798804 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.799020 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.799101 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:51.800349 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:51.800350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:51.800350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:51.800356 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:51.802054 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:51.802426 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:51.802428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:51.802428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:51.802498 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:51.802680 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:51.802681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:51.802681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:51.802861 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:51.802956 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:51.803076 (+ 120us) server_negotiation.cc:299] Negotiation successful
0621 05:24:51.803233 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 05:24:51.805200 19228 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:24:51.807289 19228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:24:51.815090 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.810771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60094 (local address 127.2.32.1:41629)
0621 05:24:51.811001 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:51.811006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:51.811229 (+ 223us) server_negotiation.cc:407] Connection header received
0621 05:24:51.811271 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:51.811273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:51.811290 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:51.811376 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:51.811615 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.812702 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.813951 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.814499 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.814589 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:51.814660 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:51.814922 (+ 262us) server_negotiation.cc:299] Negotiation successful
0621 05:24:51.815001 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":121,"threads_started":1}
I20260621 05:24:51.815063 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.810496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:41629 (local address 127.0.0.1:60094)
0621 05:24:51.811062 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:51.811074 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:51.811161 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:51.811451 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:51.811452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:51.811454 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:51.811495 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:51.811497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.812839 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.812840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:51.813860 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:51.813863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.814587 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.814588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:51.814684 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:51.814702 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 05:24:51.814879 (+ 177us) client_negotiation.cc:240] Negotiation successful
0621 05:24:51.814969 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":102,"threads_started":1}
I20260621 05:24:51.815066 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.810435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:33405 (local address 127.0.0.1:55860)
0621 05:24:51.810632 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:51.810645 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:51.810694 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:51.811313 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:51.811314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:51.811316 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:51.811389 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:51.811392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.812410 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.812411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:51.813790 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:51.813793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.814787 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.814788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:51.814888 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:51.814893 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:24:51.814928 (+ 35us) client_negotiation.cc:240] Negotiation successful
0621 05:24:51.814978 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":79,"thread_start_us":101,"threads_started":1}
I20260621 05:24:51.815573 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.810709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55860 (local address 127.2.32.3:33405)
0621 05:24:51.811080 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:51.811085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:51.811098 (+ 13us) server_negotiation.cc:407] Connection header received
0621 05:24:51.811175 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:51.811178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:51.811196 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:51.811470 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:51.811546 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.812300 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.813876 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.814677 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.815241 (+ 564us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:51.815328 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:51.815362 (+ 34us) server_negotiation.cc:299] Negotiation successful
0621 05:24:51.815410 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":118,"threads_started":1}
I20260621 05:24:51.816009 19353 tablet_service.cc:1467] Processing CreateTablet for tablet dfd11c751de44290a92fb9963ddeb918 (DEFAULT_TABLE table=test-table [id=63cea5a832c648b7a2adf4f1822f3d78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:51.816349 19353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd11c751de44290a92fb9963ddeb918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:51.816692 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.810250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:45233 (local address 127.0.0.1:52262)
0621 05:24:51.810377 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:51.810418 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:51.810483 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:51.810936 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:51.810940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:51.810944 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:51.811045 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:51.811048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.812503 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.812504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:51.814806 (+ 2302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:51.814808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.816438 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.816439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:51.816510 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:51.816515 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:24:51.816576 (+ 61us) client_negotiation.cc:240] Negotiation successful
0621 05:24:51.816620 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":974,"thread_start_us":83,"threads_started":1}
I20260621 05:24:51.816726 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.810422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.2.32.2:45233)
0621 05:24:51.810686 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:51.810692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:51.810707 (+ 15us) server_negotiation.cc:407] Connection header received
0621 05:24:51.810759 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:51.810762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:51.810782 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:51.811158 (+ 376us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:51.811251 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.812401 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.814925 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.816336 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.816425 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:51.816483 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:51.816591 (+ 108us) server_negotiation.cc:299] Negotiation successful
0621 05:24:51.816633 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":119,"threads_started":1}
I20260621 05:24:51.817446 19487 tablet_service.cc:1467] Processing CreateTablet for tablet dfd11c751de44290a92fb9963ddeb918 (DEFAULT_TABLE table=test-table [id=63cea5a832c648b7a2adf4f1822f3d78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:51.817693 19487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd11c751de44290a92fb9963ddeb918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:51.819543 19704 tablet_bootstrap.cc:492] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757: Bootstrap starting.
I20260621 05:24:51.819923 19621 tablet_service.cc:1467] Processing CreateTablet for tablet dfd11c751de44290a92fb9963ddeb918 (DEFAULT_TABLE table=test-table [id=63cea5a832c648b7a2adf4f1822f3d78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:51.820202 19621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd11c751de44290a92fb9963ddeb918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:51.820608 19704 tablet_bootstrap.cc:654] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:51.821312 19704 log.cc:824] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:51.822500 19704 tablet_bootstrap.cc:492] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757: No bootstrap required, opened a new log
I20260621 05:24:51.822571 19704 ts_tablet_manager.cc:1397] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:24:51.822945 19707 tablet_bootstrap.cc:492] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97: Bootstrap starting.
I20260621 05:24:51.821931 19706 tablet_bootstrap.cc:492] T dfd11c751de44290a92fb9963ddeb918 P 56caa4f7346a4d43bceb10693f0e73bc: Bootstrap starting.
I20260621 05:24:51.823679 19707 tablet_bootstrap.cc:654] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:51.824023 19706 tablet_bootstrap.cc:654] T dfd11c751de44290a92fb9963ddeb918 P 56caa4f7346a4d43bceb10693f0e73bc: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:51.824244 19707 log.cc:824] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:51.824326 19704 raft_consensus.cc:348] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56caa4f7346a4d43bceb10693f0e73bc" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45233 } } peers { permanent_uuid: "d004b03820ee4f62b46d7a993f326f97" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 33405 } } peers { permanent_uuid: "c6f05422b1d441909a1c6ccd75d38757" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 41629 } }
I20260621 05:24:51.824532 19704 raft_consensus.cc:374] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:51.824587 19704 raft_consensus.cc:729] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f05422b1d441909a1c6ccd75d38757, State: Initialized, Role: FOLLOWER
I20260621 05:24:51.824728 19704 consensus_queue.cc:260] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56caa4f7346a4d43bceb10693f0e73bc" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45233 } } peers { permanent_uuid: "d004b03820ee4f62b46d7a993f326f97" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 33405 } } peers { permanent_uuid: "c6f05422b1d441909a1c6ccd75d38757" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 41629 } }
I20260621 05:24:51.824796 19706 log.cc:824] T dfd11c751de44290a92fb9963ddeb918 P 56caa4f7346a4d43bceb10693f0e73bc: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:51.825112 19419 heartbeater.cc:503] Master 127.2.32.62:36785 was elected leader, sending a full tablet report...
I20260621 05:24:51.825155 19704 ts_tablet_manager.cc:1428] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 05:24:51.825301 19707 tablet_bootstrap.cc:492] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97: No bootstrap required, opened a new log
I20260621 05:24:51.825862 19706 tablet_bootstrap.cc:492] T dfd11c751de44290a92fb9963ddeb918 P 56caa4f7346a4d43bceb10693f0e73bc: No bootstrap required, opened a new log
I20260621 05:24:51.825513 19707 ts_tablet_manager.cc:1397] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:24:51.825943 19706 ts_tablet_manager.cc:1397] T dfd11c751de44290a92fb9963ddeb918 P 56caa4f7346a4d43bceb10693f0e73bc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260621 05:24:51.828068 19706 raft_consensus.cc:348] T dfd11c751de44290a92fb9963ddeb918 P 56caa4f7346a4d43bceb10693f0e73bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56caa4f7346a4d43bceb10693f0e73bc" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45233 } } peers { permanent_uuid: "d004b03820ee4f62b46d7a993f326f97" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 33405 } } peers { permanent_uuid: "c6f05422b1d441909a1c6ccd75d38757" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 41629 } }
I20260621 05:24:51.828058 19707 raft_consensus.cc:348] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56caa4f7346a4d43bceb10693f0e73bc" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45233 } } peers { permanent_uuid: "d004b03820ee4f62b46d7a993f326f97" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 33405 } } peers { permanent_uuid: "c6f05422b1d441909a1c6ccd75d38757" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 41629 } }
I20260621 05:24:51.828240 19707 raft_consensus.cc:374] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:51.828240 19706 raft_consensus.cc:374] T dfd11c751de44290a92fb9963ddeb918 P 56caa4f7346a4d43bceb10693f0e73bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:51.828276 19707 raft_consensus.cc:729] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d004b03820ee4f62b46d7a993f326f97, State: Initialized, Role: FOLLOWER
I20260621 05:24:51.828276 19706 raft_consensus.cc:729] T dfd11c751de44290a92fb9963ddeb918 P 56caa4f7346a4d43bceb10693f0e73bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56caa4f7346a4d43bceb10693f0e73bc, State: Initialized, Role: FOLLOWER
I20260621 05:24:51.828416 19707 consensus_queue.cc:260] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56caa4f7346a4d43bceb10693f0e73bc" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45233 } } peers { permanent_uuid: "d004b03820ee4f62b46d7a993f326f97" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 33405 } } peers { permanent_uuid: "c6f05422b1d441909a1c6ccd75d38757" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 41629 } }
I20260621 05:24:51.828416 19706 consensus_queue.cc:260] T dfd11c751de44290a92fb9963ddeb918 P 56caa4f7346a4d43bceb10693f0e73bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56caa4f7346a4d43bceb10693f0e73bc" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45233 } } peers { permanent_uuid: "d004b03820ee4f62b46d7a993f326f97" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 33405 } } peers { permanent_uuid: "c6f05422b1d441909a1c6ccd75d38757" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 41629 } }
I20260621 05:24:51.828698 19707 ts_tablet_manager.cc:1428] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:24:51.828698 19706 ts_tablet_manager.cc:1428] T dfd11c751de44290a92fb9963ddeb918 P 56caa4f7346a4d43bceb10693f0e73bc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:24:51.828742 19553 heartbeater.cc:503] Master 127.2.32.62:36785 was elected leader, sending a full tablet report...
I20260621 05:24:51.828824 19687 heartbeater.cc:503] Master 127.2.32.62:36785 was elected leader, sending a full tablet report...
W20260621 05:24:51.863225 19554 tablet.cc:2369] T dfd11c751de44290a92fb9963ddeb918 P 56caa4f7346a4d43bceb10693f0e73bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:24:51.922432 19709 raft_consensus.cc:482] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:24:51.922614 19709 raft_consensus.cc:504] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56caa4f7346a4d43bceb10693f0e73bc" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45233 } } peers { permanent_uuid: "d004b03820ee4f62b46d7a993f326f97" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 33405 } } peers { permanent_uuid: "c6f05422b1d441909a1c6ccd75d38757" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 41629 } }
I20260621 05:24:51.923017 19709 leader_election.cc:288] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56caa4f7346a4d43bceb10693f0e73bc (127.2.32.2:45233), d004b03820ee4f62b46d7a993f326f97 (127.2.32.3:33405)
I20260621 05:24:51.926303 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.923485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:40677 (local address 127.2.32.2:45233)
0621 05:24:51.923533 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:51.923535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:51.923637 (+ 102us) server_negotiation.cc:407] Connection header received
0621 05:24:51.923706 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:51.923708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:51.923721 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:51.923788 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:51.923960 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.924433 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.925527 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.925962 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.926037 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:51.926047 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:51.926201 (+ 154us) server_negotiation.cc:299] Negotiation successful
0621 05:24:51.926237 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:24:51.926327 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.923372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:45233 (local address 127.2.32.1:40677)
0621 05:24:51.923561 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:51.923572 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:51.923649 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:51.923793 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:51.923794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:51.923796 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:51.923885 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:51.923888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.924518 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.924519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:51.925434 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:51.925436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.926040 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.926040 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:51.926114 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:51.926118 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:24:51.926192 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 05:24:51.926234 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":111,"threads_started":1}
I20260621 05:24:51.926527 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.923373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:48327 (local address 127.2.32.3:33405)
0621 05:24:51.923464 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:51.923467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:51.923480 (+ 13us) server_negotiation.cc:407] Connection header received
0621 05:24:51.923522 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:51.923524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:51.923539 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:51.923670 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:51.923859 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.924388 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.925650 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.926162 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.926243 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:51.926254 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:51.926436 (+ 182us) server_negotiation.cc:299] Negotiation successful
0621 05:24:51.926478 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 05:24:51.926534 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.923300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:33405 (local address 127.2.32.1:48327)
0621 05:24:51.923352 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:51.923366 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:51.923423 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:51.923679 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:51.923681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:51.923683 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:51.923787 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:51.923790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.924485 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.924486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:51.925530 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:51.925535 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.926242 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.926243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:51.926344 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:51.926347 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:24:51.926403 (+ 56us) client_negotiation.cc:240] Negotiation successful
0621 05:24:51.926460 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":75}
I20260621 05:24:51.926935 19507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd11c751de44290a92fb9963ddeb918" candidate_uuid: "c6f05422b1d441909a1c6ccd75d38757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56caa4f7346a4d43bceb10693f0e73bc" is_pre_election: true
I20260621 05:24:51.926901 19641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd11c751de44290a92fb9963ddeb918" candidate_uuid: "c6f05422b1d441909a1c6ccd75d38757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d004b03820ee4f62b46d7a993f326f97" is_pre_election: true
I20260621 05:24:51.927090 19507 raft_consensus.cc:2454] T dfd11c751de44290a92fb9963ddeb918 P 56caa4f7346a4d43bceb10693f0e73bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6f05422b1d441909a1c6ccd75d38757 in term 0.
I20260621 05:24:51.927089 19641 raft_consensus.cc:2454] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6f05422b1d441909a1c6ccd75d38757 in term 0.
I20260621 05:24:51.927357 19308 leader_election.cc:302] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56caa4f7346a4d43bceb10693f0e73bc, c6f05422b1d441909a1c6ccd75d38757; no voters:
I20260621 05:24:51.927488 19709 raft_consensus.cc:2781] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:24:51.927543 19709 raft_consensus.cc:482] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:24:51.927568 19709 raft_consensus.cc:3037] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:51.928272 19709 raft_consensus.cc:504] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56caa4f7346a4d43bceb10693f0e73bc" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45233 } } peers { permanent_uuid: "d004b03820ee4f62b46d7a993f326f97" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 33405 } } peers { permanent_uuid: "c6f05422b1d441909a1c6ccd75d38757" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 41629 } }
I20260621 05:24:51.928472 19709 leader_election.cc:288] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [CANDIDATE]: Term 1 election: Requested vote from peers 56caa4f7346a4d43bceb10693f0e73bc (127.2.32.2:45233), d004b03820ee4f62b46d7a993f326f97 (127.2.32.3:33405)
I20260621 05:24:51.928604 19641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd11c751de44290a92fb9963ddeb918" candidate_uuid: "c6f05422b1d441909a1c6ccd75d38757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d004b03820ee4f62b46d7a993f326f97"
I20260621 05:24:51.928618 19507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd11c751de44290a92fb9963ddeb918" candidate_uuid: "c6f05422b1d441909a1c6ccd75d38757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56caa4f7346a4d43bceb10693f0e73bc"
I20260621 05:24:51.928684 19641 raft_consensus.cc:3037] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:51.928699 19507 raft_consensus.cc:3037] T dfd11c751de44290a92fb9963ddeb918 P 56caa4f7346a4d43bceb10693f0e73bc [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:51.929520 19641 raft_consensus.cc:2454] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6f05422b1d441909a1c6ccd75d38757 in term 1.
I20260621 05:24:51.929611 19507 raft_consensus.cc:2454] T dfd11c751de44290a92fb9963ddeb918 P 56caa4f7346a4d43bceb10693f0e73bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6f05422b1d441909a1c6ccd75d38757 in term 1.
I20260621 05:24:51.929734 19306 leader_election.cc:302] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6f05422b1d441909a1c6ccd75d38757, d004b03820ee4f62b46d7a993f326f97; no voters:
I20260621 05:24:51.929881 19709 raft_consensus.cc:2781] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:24:51.930032 19709 raft_consensus.cc:686] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [term 1 LEADER]: Becoming Leader. State: Replica: c6f05422b1d441909a1c6ccd75d38757, State: Running, Role: LEADER
I20260621 05:24:51.930146 19709 consensus_queue.cc:237] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56caa4f7346a4d43bceb10693f0e73bc" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45233 } } peers { permanent_uuid: "d004b03820ee4f62b46d7a993f326f97" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 33405 } } peers { permanent_uuid: "c6f05422b1d441909a1c6ccd75d38757" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 41629 } }
I20260621 05:24:51.931058 19227 catalog_manager.cc:5495] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6f05422b1d441909a1c6ccd75d38757 (127.2.32.1). New cstate: current_term: 1 leader_uuid: "c6f05422b1d441909a1c6ccd75d38757" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56caa4f7346a4d43bceb10693f0e73bc" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d004b03820ee4f62b46d7a993f326f97" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 33405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6f05422b1d441909a1c6ccd75d38757" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 41629 } health_report { overall_health: HEALTHY } } }
W20260621 05:24:51.946694 19420 tablet.cc:2369] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:24:51.962208 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.959333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60106 (local address 127.2.32.1:41629)
0621 05:24:51.959395 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:51.959396 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:51.959406 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:24:51.959453 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:51.959454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:51.959474 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:51.959553 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:24:51.959772 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.960262 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.961170 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:51.961353 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:51.961424 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:51.961676 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:24:51.961843 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:24:51.961937 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:51.962062 (+ 125us) server_negotiation.cc:299] Negotiation successful
0621 05:24:51.962101 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 05:24:51.964129 19507 raft_consensus.cc:1261] T dfd11c751de44290a92fb9963ddeb918 P 56caa4f7346a4d43bceb10693f0e73bc [term 1 FOLLOWER]: Refusing update from remote peer c6f05422b1d441909a1c6ccd75d38757: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:24:51.964129 19641 raft_consensus.cc:1261] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97 [term 1 FOLLOWER]: Refusing update from remote peer c6f05422b1d441909a1c6ccd75d38757: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:24:51.964522 19714 consensus_queue.cc:1035] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56caa4f7346a4d43bceb10693f0e73bc" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:24:51.964658 19709 consensus_queue.cc:1035] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d004b03820ee4f62b46d7a993f326f97" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 33405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:24:51.967726 19718 mvcc.cc:205] Tried to move back new op lower bound from 7299151839082188800 to 7299151838946402304. Current Snapshot: MvccSnapshot[applied={T|T < 7299151839082188800}]
I20260621 05:24:51.968377 19719 mvcc.cc:205] Tried to move back new op lower bound from 7299151839082188800 to 7299151838946402304. Current Snapshot: MvccSnapshot[applied={T|T < 7299151839082188800 or (T in {7299151839082188800})}]
I20260621 05:24:51.969738 19227 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:24:51.969844 19227 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:24:51.970700 19227 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 236abdc5c4b046bead7e80c2c99e36de: Sending DeleteTablet for 3 replicas of tablet dfd11c751de44290a92fb9963ddeb918
I20260621 05:24:51.971220 19621 tablet_service.cc:1514] Processing DeleteTablet for tablet dfd11c751de44290a92fb9963ddeb918 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:24:51 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:55860
I20260621 05:24:51.971251 19487 tablet_service.cc:1514] Processing DeleteTablet for tablet dfd11c751de44290a92fb9963ddeb918 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:24:51 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:52262
I20260621 05:24:51.971534 19728 tablet_replica.cc:330] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97: stopping tablet replica
I20260621 05:24:51.971643 19353 tablet_service.cc:1514] Processing DeleteTablet for tablet dfd11c751de44290a92fb9963ddeb918 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:24:51 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:60094
I20260621 05:24:51.971668 19728 raft_consensus.cc:2229] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:24:51.971824 19728 raft_consensus.cc:2258] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:24:51.971894 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 19290
I20260621 05:24:51.972002 19729 tablet_replica.cc:330] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757: stopping tablet replica
I20260621 05:24:51.972131 19729 raft_consensus.cc:2229] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:24:51.972448 19729 raft_consensus.cc:2258] T dfd11c751de44290a92fb9963ddeb918 P c6f05422b1d441909a1c6ccd75d38757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:24:51.973008 19728 ts_tablet_manager.cc:1905] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:24:51.974288 19727 tablet_replica.cc:330] T dfd11c751de44290a92fb9963ddeb918 P 56caa4f7346a4d43bceb10693f0e73bc: stopping tablet replica
I20260621 05:24:51.974426 19727 raft_consensus.cc:2229] T dfd11c751de44290a92fb9963ddeb918 P 56caa4f7346a4d43bceb10693f0e73bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:24:51.974560 19727 raft_consensus.cc:2258] T dfd11c751de44290a92fb9963ddeb918 P 56caa4f7346a4d43bceb10693f0e73bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:24:51.975464 19728 ts_tablet_manager.cc:1918] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:24:51.975553 19728 log.cc:1196] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/dfd11c751de44290a92fb9963ddeb918
I20260621 05:24:51.975879 19728 ts_tablet_manager.cc:1939] T dfd11c751de44290a92fb9963ddeb918 P d004b03820ee4f62b46d7a993f326f97: Deleting consensus metadata
I20260621 05:24:51.976428 19212 catalog_manager.cc:4841] TS d004b03820ee4f62b46d7a993f326f97 (127.2.32.3:33405): tablet dfd11c751de44290a92fb9963ddeb918 (table test-table [id=63cea5a832c648b7a2adf4f1822f3d78]) successfully deleted
W20260621 05:24:51.979954 19214 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.32.1:41629 (error 108)
I20260621 05:24:51.980159 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 19424
I20260621 05:24:51.980239 19727 ts_tablet_manager.cc:1905] T dfd11c751de44290a92fb9963ddeb918 P 56caa4f7346a4d43bceb10693f0e73bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:24:51.981487 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.980859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:41629 (local address 127.0.0.1:60114)
0621 05:24:51.981396 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:51.981452 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.1:41629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516}
W20260621 05:24:51.981716 19214 catalog_manager.cc:4584] TS c6f05422b1d441909a1c6ccd75d38757 (127.2.32.1:41629): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dfd11c751de44290a92fb9963ddeb918: Network error: Client connection negotiation failed: client connection to 127.2.32.1:41629: connect: Connection refused (error 111)
I20260621 05:24:51.985675 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:51.985512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:45233 (local address 127.0.0.1:52278)
0621 05:24:51.985583 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:51.985624 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.2:45233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260621 05:24:51.985662 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 19558
I20260621 05:24:51.990654 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 19196
2026-06-21T05:24:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2054 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-21T05:24:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:24:52Z Disabled control of system clock
I20260621 05:24:52.018467 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:46743
--webserver_interface=127.2.32.62
--webserver_port=0
--builtin_ntp_servers=127.2.32.20:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:46743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:24:52.102283 19734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:52.102488 19734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:52.102509 19734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:52.103688 19734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:24:52.103731 19734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:52.103744 19734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:52.103756 19734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:24:52.103768 19734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:24:52.105129 19734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:46743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:46743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:52.105365 19734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:52.105635 19734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:52.110324 19740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:52.110483 19739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:52.110529 19742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:52.111070 19734 server_base.cc:1034] running on GCE node
I20260621 05:24:52.111353 19734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:52.111630 19734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:52.112799 19734 hybrid_clock.cc:648] HybridClock initialized: now 1782019492112769 us; error 45 us; skew 500 ppm
I20260621 05:24:52.114578 19734 webserver.cc:460] Webserver started at http://127.2.32.62:42871/ using document root <none> and password file <none>
I20260621 05:24:52.114820 19734 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:52.114866 19734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:52.115016 19734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:52.116438 19734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "2affa463f77a46ac98e52c4002d19cda"
format_stamp: "Formatted at 2026-06-21 05:24:52 on dist-test-slave-b357"
server_key: "197cbed29e92f94e2ba68b40fbdcce63"
server_key_iv: "03debbe06140515b1c900ef14984983a"
server_key_version: "encryptionkey@0"
I20260621 05:24:52.116936 19734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "2affa463f77a46ac98e52c4002d19cda"
format_stamp: "Formatted at 2026-06-21 05:24:52 on dist-test-slave-b357"
server_key: "197cbed29e92f94e2ba68b40fbdcce63"
server_key_iv: "03debbe06140515b1c900ef14984983a"
server_key_version: "encryptionkey@0"
I20260621 05:24:52.118595 19734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:24:52.119724 19748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:52.119903 19734 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:24:52.119997 19734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "2affa463f77a46ac98e52c4002d19cda"
format_stamp: "Formatted at 2026-06-21 05:24:52 on dist-test-slave-b357"
server_key: "197cbed29e92f94e2ba68b40fbdcce63"
server_key_iv: "03debbe06140515b1c900ef14984983a"
server_key_version: "encryptionkey@0"
I20260621 05:24:52.120072 19734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:52.137996 19734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:52.138265 19734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:52.138366 19734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:52.142594 19734 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:46743
I20260621 05:24:52.142644 19800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:46743 every 8 connection(s)
I20260621 05:24:52.142970 19734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:24:52.144057 19801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:52.145160 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 19734
I20260621 05:24:52.145280 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:24:52.145514 2176 external_mini_cluster.cc:1428] Setting key 335694f8b4b8d364018ca16ad1f6e449
I20260621 05:24:52.147013 19801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda: Bootstrap starting.
I20260621 05:24:52.147732 19801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:52.148046 19801 log.cc:824] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:52.148909 19801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda: No bootstrap required, opened a new log
I20260621 05:24:52.150955 19801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2affa463f77a46ac98e52c4002d19cda" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46743 } }
I20260621 05:24:52.151104 19801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:52.151175 19801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2affa463f77a46ac98e52c4002d19cda, State: Initialized, Role: FOLLOWER
I20260621 05:24:52.151326 19801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2affa463f77a46ac98e52c4002d19cda" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46743 } }
I20260621 05:24:52.151404 19801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:24:52.151446 19801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:24:52.151512 19801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:52.152240 19801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2affa463f77a46ac98e52c4002d19cda" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46743 } }
I20260621 05:24:52.152339 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:52.146166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48106 (local address 127.2.32.62:46743)
0621 05:24:52.146414 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:52.146424 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:52.146459 (+ 35us) server_negotiation.cc:407] Connection header received
0621 05:24:52.147000 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:52.147009 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:52.147086 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:52.147284 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:52.147632 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.148847 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.149784 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.150405 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.150565 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:52.151552 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:52.151556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:52.151556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:24:52.151764 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:52.151806 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:24:52.151820 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:52.151901 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:52.152037 (+ 136us) server_negotiation.cc:299] Negotiation successful
0621 05:24:52.152117 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":107,"threads_started":1}
I20260621 05:24:52.152343 19801 leader_election.cc:302] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2affa463f77a46ac98e52c4002d19cda; no voters:
I20260621 05:24:52.152738 19801 leader_election.cc:288] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:24:52.152760 19806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:24:52.152967 19806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda [term 1 LEADER]: Becoming Leader. State: Replica: 2affa463f77a46ac98e52c4002d19cda, State: Running, Role: LEADER
I20260621 05:24:52.153108 19801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:24:52.153093 19806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2affa463f77a46ac98e52c4002d19cda" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46743 } }
I20260621 05:24:52.153523 19806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2affa463f77a46ac98e52c4002d19cda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2affa463f77a46ac98e52c4002d19cda" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46743 } } }
I20260621 05:24:52.153620 19806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda [sys.catalog]: This master's current role is: LEADER
I20260621 05:24:52.153782 19806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2affa463f77a46ac98e52c4002d19cda. Latest consensus state: current_term: 1 leader_uuid: "2affa463f77a46ac98e52c4002d19cda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2affa463f77a46ac98e52c4002d19cda" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46743 } } }
I20260621 05:24:52.153864 19806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda [sys.catalog]: This master's current role is: LEADER
I20260621 05:24:52.154301 19812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:24:52.154906 19812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:24:52.156862 19812 catalog_manager.cc:1347] Generated new cluster ID: b5c12863f2b8493da0a9c251300e6cb2
I20260621 05:24:52.156944 19812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:24:52.171700 19812 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:24:52.172255 19812 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:24:52.175078 19812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda: Generated new TSK 0
I20260621 05:24:52.175328 19812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:24:52.180720 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46743
--builtin_ntp_servers=127.2.32.20:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:24:52.278941 19825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:52.279214 19825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:52.279258 19825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:52.280437 19825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:52.280476 19825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:52.280503 19825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:24:52.281862 19825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:52.282107 19825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:52.282418 19825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:52.288235 19833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:52.288214 19830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:52.288215 19831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:52.289149 19825 server_base.cc:1034] running on GCE node
I20260621 05:24:52.289453 19825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:52.289780 19825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:52.290994 19825 hybrid_clock.cc:648] HybridClock initialized: now 1782019492290966 us; error 47 us; skew 500 ppm
I20260621 05:24:52.292697 19825 webserver.cc:460] Webserver started at http://127.2.32.1:43809/ using document root <none> and password file <none>
I20260621 05:24:52.293015 19825 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:52.293105 19825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:52.293277 19825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:52.294611 19825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "e6f1c8b21fec4ce5ac34e7cd6a4fe6d5"
format_stamp: "Formatted at 2026-06-21 05:24:52 on dist-test-slave-b357"
server_key: "fb12985e4affa57ad8c146d244649802"
server_key_iv: "e812fc9cb71e8ee0c7af2514e3ee5db1"
server_key_version: "encryptionkey@0"
I20260621 05:24:52.295029 19825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "e6f1c8b21fec4ce5ac34e7cd6a4fe6d5"
format_stamp: "Formatted at 2026-06-21 05:24:52 on dist-test-slave-b357"
server_key: "fb12985e4affa57ad8c146d244649802"
server_key_iv: "e812fc9cb71e8ee0c7af2514e3ee5db1"
server_key_version: "encryptionkey@0"
I20260621 05:24:52.296868 19825 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260621 05:24:52.297729 19839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:52.297927 19825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:52.298038 19825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "e6f1c8b21fec4ce5ac34e7cd6a4fe6d5"
format_stamp: "Formatted at 2026-06-21 05:24:52 on dist-test-slave-b357"
server_key: "fb12985e4affa57ad8c146d244649802"
server_key_iv: "e812fc9cb71e8ee0c7af2514e3ee5db1"
server_key_version: "encryptionkey@0"
I20260621 05:24:52.298146 19825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:52.308365 19825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:52.308588 19825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:52.308688 19825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:52.308849 19825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:52.309182 19825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:52.309221 19825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:52.309242 19825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:52.309255 19825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:52.315099 19825 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:34873
I20260621 05:24:52.315208 19952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:34873 every 8 connection(s)
I20260621 05:24:52.315533 19825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:24:52.317754 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 19825
I20260621 05:24:52.317847 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:24:52.318070 2176 external_mini_cluster.cc:1428] Setting key d138b27460d58f50f2eb6cf86e4eb228
I20260621 05:24:52.319005 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46743
--builtin_ntp_servers=127.2.32.20:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 05:24:52.321446 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:52.316366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:34113 (local address 127.2.32.62:46743)
0621 05:24:52.316458 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:52.316460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:52.316652 (+ 192us) server_negotiation.cc:407] Connection header received
0621 05:24:52.317129 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:52.317130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:52.317142 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:52.317207 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:52.317906 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.318472 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.319643 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.319846 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.319940 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:52.320984 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:52.320987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:52.320987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:24:52.320995 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:52.321017 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:24:52.321023 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:52.321190 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:52.321301 (+ 111us) server_negotiation.cc:299] Negotiation successful
0621 05:24:52.321329 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 05:24:52.321616 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:52.316330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:46743 (local address 127.2.32.1:34113)
0621 05:24:52.316511 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:52.316551 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:52.317034 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:52.317274 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:52.317277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:52.317285 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:52.317798 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:52.317822 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.318561 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.318562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:52.319561 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:52.319563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.319951 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.319952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:52.320083 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:52.320733 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:24:52.320734 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:52.320779 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:24:52.320816 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:52.321200 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:52.321207 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:52.321210 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:24:52.321294 (+ 84us) client_negotiation.cc:240] Negotiation successful
0621 05:24:52.321382 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260621 05:24:52.322237 19953 heartbeater.cc:347] Connected to a master server at 127.2.32.62:46743
I20260621 05:24:52.322393 19953 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:52.322652 19953 heartbeater.cc:511] Master 127.2.32.62:46743 requested a full tablet report, sending...
I20260621 05:24:52.323393 19765 ts_manager.cc:194] Registered new tserver with Master: e6f1c8b21fec4ce5ac34e7cd6a4fe6d5 (127.2.32.1:34873)
I20260621 05:24:52.324223 19765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.32.1:34113
W20260621 05:24:52.400496 19956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:52.400729 19956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:52.400765 19956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:52.402107 19956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:52.402170 19956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:52.402218 19956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:24:52.404156 19956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:52.404438 19956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:52.404712 19956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:52.409906 19962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:52.409899 19964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:52.409895 19961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:52.410118 19956 server_base.cc:1034] running on GCE node
I20260621 05:24:52.410358 19956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:52.410609 19956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:52.411837 19956 hybrid_clock.cc:648] HybridClock initialized: now 1782019492411828 us; error 47 us; skew 500 ppm
I20260621 05:24:52.413208 19956 webserver.cc:460] Webserver started at http://127.2.32.2:34513/ using document root <none> and password file <none>
I20260621 05:24:52.413470 19956 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:52.413540 19956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:52.413693 19956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:52.414749 19956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "496f53d6fe1242efb3aa8201df828438"
format_stamp: "Formatted at 2026-06-21 05:24:52 on dist-test-slave-b357"
server_key: "56d6893c7c476cd6b80a45722c9ba779"
server_key_iv: "b368cddff99162a50acebae206bba3ca"
server_key_version: "encryptionkey@0"
I20260621 05:24:52.415107 19956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "496f53d6fe1242efb3aa8201df828438"
format_stamp: "Formatted at 2026-06-21 05:24:52 on dist-test-slave-b357"
server_key: "56d6893c7c476cd6b80a45722c9ba779"
server_key_iv: "b368cddff99162a50acebae206bba3ca"
server_key_version: "encryptionkey@0"
I20260621 05:24:52.416446 19956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:52.417193 19970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:52.417464 19956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:52.417558 19956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "496f53d6fe1242efb3aa8201df828438"
format_stamp: "Formatted at 2026-06-21 05:24:52 on dist-test-slave-b357"
server_key: "56d6893c7c476cd6b80a45722c9ba779"
server_key_iv: "b368cddff99162a50acebae206bba3ca"
server_key_version: "encryptionkey@0"
I20260621 05:24:52.417627 19956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:52.429194 19956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:52.429608 19956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:52.429776 19956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:52.430054 19956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:52.430428 19956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:52.430480 19956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:52.430526 19956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:52.430557 19956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:52.438186 19956 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:34859
I20260621 05:24:52.438230 20083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:34859 every 8 connection(s)
I20260621 05:24:52.438573 19956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:24:52.444211 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:52.439372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:46267 (local address 127.2.32.62:46743)
0621 05:24:52.439431 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:52.439433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:52.439670 (+ 237us) server_negotiation.cc:407] Connection header received
0621 05:24:52.440136 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:52.440138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:52.440148 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:52.440211 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:52.440996 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.441486 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.442586 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.442787 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.442864 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:52.443859 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:52.443862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:52.443862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:24:52.443870 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:52.443896 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:24:52.443902 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:52.444000 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:52.444120 (+ 120us) server_negotiation.cc:299] Negotiation successful
0621 05:24:52.444144 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 05:24:52.444516 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 19956
I20260621 05:24:52.444630 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:24:52.444832 2176 external_mini_cluster.cc:1428] Setting key 7cfca316566d46fc92206f5806b18d53
I20260621 05:24:52.445060 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:52.439360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:46743 (local address 127.2.32.2:46267)
0621 05:24:52.439535 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:52.439569 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:52.440043 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:52.440277 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:52.440280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:52.440287 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:52.440890 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:52.440896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.441591 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.441592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:52.442468 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:52.442471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.442901 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.442901 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:52.443034 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:52.443668 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:24:52.443668 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:52.443696 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:24:52.443717 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:52.444012 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:52.444016 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:52.444019 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:24:52.444073 (+ 54us) client_negotiation.cc:240] Negotiation successful
0621 05:24:52.444151 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260621 05:24:52.445713 20084 heartbeater.cc:347] Connected to a master server at 127.2.32.62:46743
I20260621 05:24:52.445806 20084 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:52.445995 20084 heartbeater.cc:511] Master 127.2.32.62:46743 requested a full tablet report, sending...
I20260621 05:24:52.446168 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46743
--builtin_ntp_servers=127.2.32.20:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 05:24:52.446398 19765 ts_manager.cc:194] Registered new tserver with Master: 496f53d6fe1242efb3aa8201df828438 (127.2.32.2:34859)
I20260621 05:24:52.446780 19765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.32.2:46267
W20260621 05:24:52.536058 20087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:52.536343 20087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:52.536384 20087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:52.537663 20087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:52.537732 20087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:52.537774 20087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:24:52.539465 20087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:52.539810 20087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:52.540151 20087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:52.545596 20095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:52.545622 20093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:52.545676 20092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:52.546077 20087 server_base.cc:1034] running on GCE node
I20260621 05:24:52.546303 20087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:52.546563 20087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:52.547756 20087 hybrid_clock.cc:648] HybridClock initialized: now 1782019492547752 us; error 42 us; skew 500 ppm
I20260621 05:24:52.549365 20087 webserver.cc:460] Webserver started at http://127.2.32.3:42835/ using document root <none> and password file <none>
I20260621 05:24:52.549646 20087 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:52.549726 20087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:52.549885 20087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:52.551070 20087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "0dbccd4274b04fb9b5d9f125d919640f"
format_stamp: "Formatted at 2026-06-21 05:24:52 on dist-test-slave-b357"
server_key: "f8ace22dbb1b0cc9ac70db846ecd66b6"
server_key_iv: "aec6b0cad258d9c389e044d86f2409de"
server_key_version: "encryptionkey@0"
I20260621 05:24:52.551548 20087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "0dbccd4274b04fb9b5d9f125d919640f"
format_stamp: "Formatted at 2026-06-21 05:24:52 on dist-test-slave-b357"
server_key: "f8ace22dbb1b0cc9ac70db846ecd66b6"
server_key_iv: "aec6b0cad258d9c389e044d86f2409de"
server_key_version: "encryptionkey@0"
I20260621 05:24:52.553254 20087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:24:52.554234 20101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:52.554486 20087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:52.554606 20087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "0dbccd4274b04fb9b5d9f125d919640f"
format_stamp: "Formatted at 2026-06-21 05:24:52 on dist-test-slave-b357"
server_key: "f8ace22dbb1b0cc9ac70db846ecd66b6"
server_key_iv: "aec6b0cad258d9c389e044d86f2409de"
server_key_version: "encryptionkey@0"
I20260621 05:24:52.554677 20087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:52.593386 20087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:52.593690 20087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:52.593799 20087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:52.594009 20087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:52.594380 20087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:52.594414 20087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:52.594468 20087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:52.594496 20087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:52.600165 20087 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:41319
I20260621 05:24:52.600226 20214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:41319 every 8 connection(s)
I20260621 05:24:52.600543 20087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:24:52.602823 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 20087
I20260621 05:24:52.602936 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:24:52.603229 2176 external_mini_cluster.cc:1428] Setting key d286c807913126e3865af1ae44e74c9c
I20260621 05:24:52.606832 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:52.601468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:46611 (local address 127.2.32.62:46743)
0621 05:24:52.601542 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:52.601544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:52.601683 (+ 139us) server_negotiation.cc:407] Connection header received
0621 05:24:52.602196 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:52.602198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:52.602211 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:52.602309 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:52.603220 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.604036 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.605301 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.605494 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.605570 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:52.606501 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:52.606503 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:52.606504 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:24:52.606510 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:52.606532 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:24:52.606537 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:52.606623 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:52.606738 (+ 115us) server_negotiation.cc:299] Negotiation successful
0621 05:24:52.606763 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 05:24:52.606973 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:52.601336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:46743 (local address 127.2.32.3:46611)
0621 05:24:52.601518 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:52.601558 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:52.602050 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:52.602404 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:52.602408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:52.602415 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:52.603096 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:52.603100 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.604119 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.604120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:52.605169 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:52.605176 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.605619 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.605620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:52.605752 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:52.606336 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:24:52.606336 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:52.606365 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:24:52.606387 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:52.606635 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:52.606639 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:52.606641 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:24:52.606699 (+ 58us) client_negotiation.cc:240] Negotiation successful
0621 05:24:52.606775 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260621 05:24:52.607466 20215 heartbeater.cc:347] Connected to a master server at 127.2.32.62:46743
I20260621 05:24:52.607560 20215 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:52.607731 20215 heartbeater.cc:511] Master 127.2.32.62:46743 requested a full tablet report, sending...
I20260621 05:24:52.608089 19765 ts_manager.cc:194] Registered new tserver with Master: 0dbccd4274b04fb9b5d9f125d919640f (127.2.32.3:41319)
I20260621 05:24:52.608462 19765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.32.3:46611
I20260621 05:24:52.614770 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:24:52.618938 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:52.616125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48108 (local address 127.2.32.62:46743)
0621 05:24:52.616191 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:52.616193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:52.616308 (+ 115us) server_negotiation.cc:407] Connection header received
0621 05:24:52.616397 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:52.616398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:52.616408 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:52.616458 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:52.616613 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.617064 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.617797 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.618001 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.618082 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:52.618705 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:52.618706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:52.618707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:24:52.618711 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:52.618725 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:24:52.618729 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:52.618782 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:52.618841 (+ 59us) server_negotiation.cc:299] Negotiation successful
0621 05:24:52.618859 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 05:24:52.620587 19765 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:24:52.622237 19765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:24:52.629498 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:52.625363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45422 (local address 127.2.32.3:41319)
0621 05:24:52.625750 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:52.625753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:52.625761 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:24:52.625788 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:52.625790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:52.625800 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:52.625894 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:52.626115 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.626817 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.628596 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.629128 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.629195 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:52.629254 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:52.629412 (+ 158us) server_negotiation.cc:299] Negotiation successful
0621 05:24:52.629439 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":100,"threads_started":1}
I20260621 05:24:52.629904 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:52.624985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:41319 (local address 127.0.0.1:45422)
0621 05:24:52.625571 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:52.625581 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:52.625630 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:52.625988 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:52.625989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:52.625993 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:52.626056 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:52.626058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.626902 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.626903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:52.628453 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:52.628456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.629214 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.629215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:52.629294 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:52.629310 (+ 16us) client_negotiation.cc:769] Sending connection context
0621 05:24:52.629400 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 05:24:52.629453 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":106,"threads_started":1}
I20260621 05:24:52.630605 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:52.624866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:34859 (local address 127.0.0.1:37742)
0621 05:24:52.625215 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:52.625272 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:52.625370 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:52.625895 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:52.625899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:52.625902 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:52.625989 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:52.625991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.628671 (+ 2680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.628672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:52.629494 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:52.629496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.630390 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.630391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:52.630455 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:52.630459 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:24:52.630499 (+ 40us) client_negotiation.cc:240] Negotiation successful
0621 05:24:52.630539 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":83,"threads_started":1}
I20260621 05:24:52.630897 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:52.624985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:34873 (local address 127.0.0.1:40962)
0621 05:24:52.625781 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:52.625790 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:52.625850 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:52.626040 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:52.626041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:52.626044 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:52.626092 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:52.626094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.627497 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.627498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:52.628573 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:52.628575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.630654 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.630655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:52.630716 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:52.630721 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:24:52.630776 (+ 55us) client_negotiation.cc:240] Negotiation successful
0621 05:24:52.630827 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":2,"thread_start_us":104,"threads_started":1}
I20260621 05:24:52.630924 20149 tablet_service.cc:1467] Processing CreateTablet for tablet 98236352a4954ac1a96e1367425e9368 (DEFAULT_TABLE table=test-table [id=c629e15e0c8540be99225482e67ab29b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:52.631178 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:52.625263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40962 (local address 127.2.32.1:34873)
0621 05:24:52.625482 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:52.625485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:52.625870 (+ 385us) server_negotiation.cc:407] Connection header received
0621 05:24:52.625929 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:52.625932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:52.625946 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:52.626024 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:52.626247 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.627384 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.629608 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.630446 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.630517 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:52.630616 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:52.631021 (+ 405us) server_negotiation.cc:299] Negotiation successful
0621 05:24:52.631070 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":100,"threads_started":1}
I20260621 05:24:52.631341 20149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98236352a4954ac1a96e1367425e9368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:52.631819 19887 tablet_service.cc:1467] Processing CreateTablet for tablet 98236352a4954ac1a96e1367425e9368 (DEFAULT_TABLE table=test-table [id=c629e15e0c8540be99225482e67ab29b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:52.632179 19887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98236352a4954ac1a96e1367425e9368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:52.634845 20229 tablet_bootstrap.cc:492] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f: Bootstrap starting.
I20260621 05:24:52.635030 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:52.624947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37742 (local address 127.2.32.2:34859)
0621 05:24:52.625141 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:52.625146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:52.625366 (+ 220us) server_negotiation.cc:407] Connection header received
0621 05:24:52.625551 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:52.625554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:52.625574 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:52.625648 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:52.626120 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.627391 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.629586 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.630302 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.630379 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:52.630482 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:52.634900 (+ 4418us) server_negotiation.cc:299] Negotiation successful
0621 05:24:52.634950 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":126,"threads_started":1}
I20260621 05:24:52.635676 20229 tablet_bootstrap.cc:654] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:52.635766 20018 tablet_service.cc:1467] Processing CreateTablet for tablet 98236352a4954ac1a96e1367425e9368 (DEFAULT_TABLE table=test-table [id=c629e15e0c8540be99225482e67ab29b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:52.636099 20230 tablet_bootstrap.cc:492] T 98236352a4954ac1a96e1367425e9368 P e6f1c8b21fec4ce5ac34e7cd6a4fe6d5: Bootstrap starting.
I20260621 05:24:52.636164 20018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98236352a4954ac1a96e1367425e9368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:52.636224 20229 log.cc:824] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:52.636878 20230 tablet_bootstrap.cc:654] T 98236352a4954ac1a96e1367425e9368 P e6f1c8b21fec4ce5ac34e7cd6a4fe6d5: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:52.637161 20229 tablet_bootstrap.cc:492] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f: No bootstrap required, opened a new log
I20260621 05:24:52.637236 20229 ts_tablet_manager.cc:1397] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:24:52.637353 20230 log.cc:824] T 98236352a4954ac1a96e1367425e9368 P e6f1c8b21fec4ce5ac34e7cd6a4fe6d5: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:52.638289 20230 tablet_bootstrap.cc:492] T 98236352a4954ac1a96e1367425e9368 P e6f1c8b21fec4ce5ac34e7cd6a4fe6d5: No bootstrap required, opened a new log
I20260621 05:24:52.638419 20230 ts_tablet_manager.cc:1397] T 98236352a4954ac1a96e1367425e9368 P e6f1c8b21fec4ce5ac34e7cd6a4fe6d5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:24:52.639665 20233 tablet_bootstrap.cc:492] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438: Bootstrap starting.
I20260621 05:24:52.639894 20229 raft_consensus.cc:348] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f1c8b21fec4ce5ac34e7cd6a4fe6d5" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 34873 } } peers { permanent_uuid: "496f53d6fe1242efb3aa8201df828438" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 34859 } } peers { permanent_uuid: "0dbccd4274b04fb9b5d9f125d919640f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41319 } }
I20260621 05:24:52.640061 20229 raft_consensus.cc:374] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:52.640098 20229 raft_consensus.cc:729] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dbccd4274b04fb9b5d9f125d919640f, State: Initialized, Role: FOLLOWER
I20260621 05:24:52.640276 20233 tablet_bootstrap.cc:654] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:52.640560 20233 log.cc:824] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:52.640234 20229 consensus_queue.cc:260] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f1c8b21fec4ce5ac34e7cd6a4fe6d5" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 34873 } } peers { permanent_uuid: "496f53d6fe1242efb3aa8201df828438" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 34859 } } peers { permanent_uuid: "0dbccd4274b04fb9b5d9f125d919640f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41319 } }
I20260621 05:24:52.640789 20229 ts_tablet_manager.cc:1428] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260621 05:24:52.640916 20215 heartbeater.cc:503] Master 127.2.32.62:46743 was elected leader, sending a full tablet report...
I20260621 05:24:52.641268 20233 tablet_bootstrap.cc:492] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438: No bootstrap required, opened a new log
I20260621 05:24:52.641357 20233 ts_tablet_manager.cc:1397] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:24:52.641295 20230 raft_consensus.cc:348] T 98236352a4954ac1a96e1367425e9368 P e6f1c8b21fec4ce5ac34e7cd6a4fe6d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f1c8b21fec4ce5ac34e7cd6a4fe6d5" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 34873 } } peers { permanent_uuid: "496f53d6fe1242efb3aa8201df828438" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 34859 } } peers { permanent_uuid: "0dbccd4274b04fb9b5d9f125d919640f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41319 } }
I20260621 05:24:52.641448 20230 raft_consensus.cc:374] T 98236352a4954ac1a96e1367425e9368 P e6f1c8b21fec4ce5ac34e7cd6a4fe6d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:52.641487 20230 raft_consensus.cc:729] T 98236352a4954ac1a96e1367425e9368 P e6f1c8b21fec4ce5ac34e7cd6a4fe6d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f1c8b21fec4ce5ac34e7cd6a4fe6d5, State: Initialized, Role: FOLLOWER
I20260621 05:24:52.641608 20230 consensus_queue.cc:260] T 98236352a4954ac1a96e1367425e9368 P e6f1c8b21fec4ce5ac34e7cd6a4fe6d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f1c8b21fec4ce5ac34e7cd6a4fe6d5" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 34873 } } peers { permanent_uuid: "496f53d6fe1242efb3aa8201df828438" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 34859 } } peers { permanent_uuid: "0dbccd4274b04fb9b5d9f125d919640f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41319 } }
I20260621 05:24:52.641880 20230 ts_tablet_manager.cc:1428] T 98236352a4954ac1a96e1367425e9368 P e6f1c8b21fec4ce5ac34e7cd6a4fe6d5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:24:52.641940 19953 heartbeater.cc:503] Master 127.2.32.62:46743 was elected leader, sending a full tablet report...
I20260621 05:24:52.643787 20233 raft_consensus.cc:348] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f1c8b21fec4ce5ac34e7cd6a4fe6d5" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 34873 } } peers { permanent_uuid: "496f53d6fe1242efb3aa8201df828438" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 34859 } } peers { permanent_uuid: "0dbccd4274b04fb9b5d9f125d919640f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41319 } }
I20260621 05:24:52.643920 20233 raft_consensus.cc:374] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:52.643962 20233 raft_consensus.cc:729] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 496f53d6fe1242efb3aa8201df828438, State: Initialized, Role: FOLLOWER
I20260621 05:24:52.644064 20233 consensus_queue.cc:260] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f1c8b21fec4ce5ac34e7cd6a4fe6d5" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 34873 } } peers { permanent_uuid: "496f53d6fe1242efb3aa8201df828438" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 34859 } } peers { permanent_uuid: "0dbccd4274b04fb9b5d9f125d919640f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41319 } }
I20260621 05:24:52.644290 20233 ts_tablet_manager.cc:1428] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:24:52.644368 20084 heartbeater.cc:503] Master 127.2.32.62:46743 was elected leader, sending a full tablet report...
W20260621 05:24:52.689195 20085 tablet.cc:2369] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 05:24:52.816284 19954 tablet.cc:2369] T 98236352a4954ac1a96e1367425e9368 P e6f1c8b21fec4ce5ac34e7cd6a4fe6d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:24:52.828009 20237 raft_consensus.cc:482] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:24:52.828195 20237 raft_consensus.cc:504] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f1c8b21fec4ce5ac34e7cd6a4fe6d5" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 34873 } } peers { permanent_uuid: "496f53d6fe1242efb3aa8201df828438" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 34859 } } peers { permanent_uuid: "0dbccd4274b04fb9b5d9f125d919640f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41319 } }
I20260621 05:24:52.828701 20237 leader_election.cc:288] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6f1c8b21fec4ce5ac34e7cd6a4fe6d5 (127.2.32.1:34873), 0dbccd4274b04fb9b5d9f125d919640f (127.2.32.3:41319)
I20260621 05:24:52.832504 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:52.828961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:34873 (local address 127.2.32.2:46555)
0621 05:24:52.829020 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:52.829035 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:52.829124 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:52.829368 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:52.829369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:52.829372 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:52.829459 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:52.829461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.830261 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.830263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:52.831380 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:52.831383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.832157 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.832158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:52.832250 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:52.832255 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:24:52.832348 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 05:24:52.832405 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260621 05:24:52.832527 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:52.829033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:46555 (local address 127.2.32.1:34873)
0621 05:24:52.829099 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:52.829102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:52.829116 (+ 14us) server_negotiation.cc:407] Connection header received
0621 05:24:52.829260 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:52.829261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:52.829270 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:52.829338 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:52.829565 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.830161 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.831515 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.832045 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.832118 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:52.832129 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:52.832364 (+ 235us) server_negotiation.cc:299] Negotiation successful
0621 05:24:52.832453 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260621 05:24:52.832779 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:52.829055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:34585 (local address 127.2.32.3:41319)
0621 05:24:52.829138 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:52.829142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:52.829540 (+ 398us) server_negotiation.cc:407] Connection header received
0621 05:24:52.829617 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:52.829618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:52.829629 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:52.829676 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:52.829911 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.830579 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.831889 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.832367 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.832436 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:52.832446 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:52.832670 (+ 224us) server_negotiation.cc:299] Negotiation successful
0621 05:24:52.832717 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 05:24:52.832857 19907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98236352a4954ac1a96e1367425e9368" candidate_uuid: "496f53d6fe1242efb3aa8201df828438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f1c8b21fec4ce5ac34e7cd6a4fe6d5" is_pre_election: true
I20260621 05:24:52.833009 20238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:52.828947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:41319 (local address 127.2.32.2:34585)
0621 05:24:52.829450 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:52.829461 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:52.829539 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:52.829701 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:52.829701 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:52.829703 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:52.829787 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:52.829790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.830713 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.830714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:52.831772 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:52.831774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.832465 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.832466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:52.832556 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:52.832561 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:24:52.832883 (+ 322us) client_negotiation.cc:240] Negotiation successful
0621 05:24:52.832937 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":225,"thread_start_us":152,"threads_started":1}
I20260621 05:24:52.833015 19907 raft_consensus.cc:2454] T 98236352a4954ac1a96e1367425e9368 P e6f1c8b21fec4ce5ac34e7cd6a4fe6d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 496f53d6fe1242efb3aa8201df828438 in term 0.
I20260621 05:24:52.833267 19973 leader_election.cc:302] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 496f53d6fe1242efb3aa8201df828438, e6f1c8b21fec4ce5ac34e7cd6a4fe6d5; no voters:
I20260621 05:24:52.833405 20169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98236352a4954ac1a96e1367425e9368" candidate_uuid: "496f53d6fe1242efb3aa8201df828438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dbccd4274b04fb9b5d9f125d919640f" is_pre_election: true
I20260621 05:24:52.833467 20237 raft_consensus.cc:2781] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:24:52.833528 20237 raft_consensus.cc:482] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:24:52.833551 20237 raft_consensus.cc:3037] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:52.833554 20169 raft_consensus.cc:2454] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 496f53d6fe1242efb3aa8201df828438 in term 0.
I20260621 05:24:52.834296 20237 raft_consensus.cc:504] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f1c8b21fec4ce5ac34e7cd6a4fe6d5" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 34873 } } peers { permanent_uuid: "496f53d6fe1242efb3aa8201df828438" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 34859 } } peers { permanent_uuid: "0dbccd4274b04fb9b5d9f125d919640f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41319 } }
I20260621 05:24:52.834446 20237 leader_election.cc:288] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [CANDIDATE]: Term 1 election: Requested vote from peers e6f1c8b21fec4ce5ac34e7cd6a4fe6d5 (127.2.32.1:34873), 0dbccd4274b04fb9b5d9f125d919640f (127.2.32.3:41319)
I20260621 05:24:52.834620 20169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98236352a4954ac1a96e1367425e9368" candidate_uuid: "496f53d6fe1242efb3aa8201df828438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dbccd4274b04fb9b5d9f125d919640f"
I20260621 05:24:52.834617 19907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98236352a4954ac1a96e1367425e9368" candidate_uuid: "496f53d6fe1242efb3aa8201df828438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f1c8b21fec4ce5ac34e7cd6a4fe6d5"
I20260621 05:24:52.834686 19907 raft_consensus.cc:3037] T 98236352a4954ac1a96e1367425e9368 P e6f1c8b21fec4ce5ac34e7cd6a4fe6d5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:52.834704 20169 raft_consensus.cc:3037] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:52.835409 20169 raft_consensus.cc:2454] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 496f53d6fe1242efb3aa8201df828438 in term 1.
I20260621 05:24:52.835564 19907 raft_consensus.cc:2454] T 98236352a4954ac1a96e1367425e9368 P e6f1c8b21fec4ce5ac34e7cd6a4fe6d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 496f53d6fe1242efb3aa8201df828438 in term 1.
I20260621 05:24:52.835625 19971 leader_election.cc:302] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dbccd4274b04fb9b5d9f125d919640f, 496f53d6fe1242efb3aa8201df828438; no voters:
I20260621 05:24:52.835708 20237 raft_consensus.cc:2781] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:24:52.835860 20237 raft_consensus.cc:686] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [term 1 LEADER]: Becoming Leader. State: Replica: 496f53d6fe1242efb3aa8201df828438, State: Running, Role: LEADER
I20260621 05:24:52.835952 20237 consensus_queue.cc:237] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f1c8b21fec4ce5ac34e7cd6a4fe6d5" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 34873 } } peers { permanent_uuid: "496f53d6fe1242efb3aa8201df828438" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 34859 } } peers { permanent_uuid: "0dbccd4274b04fb9b5d9f125d919640f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41319 } }
I20260621 05:24:52.836740 19765 catalog_manager.cc:5495] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 reported cstate change: term changed from 0 to 1, leader changed from <none> to 496f53d6fe1242efb3aa8201df828438 (127.2.32.2). New cstate: current_term: 1 leader_uuid: "496f53d6fe1242efb3aa8201df828438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f1c8b21fec4ce5ac34e7cd6a4fe6d5" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 34873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "496f53d6fe1242efb3aa8201df828438" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 34859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dbccd4274b04fb9b5d9f125d919640f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41319 } health_report { overall_health: UNKNOWN } } }
W20260621 05:24:52.851074 20216 tablet.cc:2369] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:24:52.856544 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:52.854033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.2.32.2:34859)
0621 05:24:52.854089 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:52.854091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:52.854105 (+ 14us) server_negotiation.cc:407] Connection header received
0621 05:24:52.854162 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:52.854163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:52.854180 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:52.854239 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:24:52.854402 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.854919 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.855786 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:52.855974 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:52.856034 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:52.856205 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:24:52.856304 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:24:52.856385 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:52.856451 (+ 66us) server_negotiation.cc:299] Negotiation successful
0621 05:24:52.856468 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 05:24:52.858752 20169 raft_consensus.cc:1261] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f [term 1 FOLLOWER]: Refusing update from remote peer 496f53d6fe1242efb3aa8201df828438: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:24:52.858868 19907 raft_consensus.cc:1261] T 98236352a4954ac1a96e1367425e9368 P e6f1c8b21fec4ce5ac34e7cd6a4fe6d5 [term 1 FOLLOWER]: Refusing update from remote peer 496f53d6fe1242efb3aa8201df828438: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:24:52.859242 20239 consensus_queue.cc:1035] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f1c8b21fec4ce5ac34e7cd6a4fe6d5" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 34873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:24:52.859381 20237 consensus_queue.cc:1035] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dbccd4274b04fb9b5d9f125d919640f" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 41319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:24:52.861682 20242 mvcc.cc:205] Tried to move back new op lower bound from 7299151842746380288 to 7299151842656583680. Current Snapshot: MvccSnapshot[applied={T|T < 7299151842746380288}]
I20260621 05:24:52.863859 19764 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:24:52.863953 19764 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:24:52.864725 19764 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2affa463f77a46ac98e52c4002d19cda: Sending DeleteTablet for 3 replicas of tablet 98236352a4954ac1a96e1367425e9368
I20260621 05:24:52.865020 19887 tablet_service.cc:1514] Processing DeleteTablet for tablet 98236352a4954ac1a96e1367425e9368 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:24:52 UTC) from {username='slave'} at 127.0.0.1:40962
I20260621 05:24:52.865155 20018 tablet_service.cc:1514] Processing DeleteTablet for tablet 98236352a4954ac1a96e1367425e9368 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:24:52 UTC) from {username='slave'} at 127.0.0.1:37742
I20260621 05:24:52.865231 20252 tablet_replica.cc:330] T 98236352a4954ac1a96e1367425e9368 P e6f1c8b21fec4ce5ac34e7cd6a4fe6d5: stopping tablet replica
I20260621 05:24:52.865224 20149 tablet_service.cc:1514] Processing DeleteTablet for tablet 98236352a4954ac1a96e1367425e9368 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:24:52 UTC) from {username='slave'} at 127.0.0.1:45422
I20260621 05:24:52.865317 20252 raft_consensus.cc:2229] T 98236352a4954ac1a96e1367425e9368 P e6f1c8b21fec4ce5ac34e7cd6a4fe6d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:24:52.865425 20254 tablet_replica.cc:330] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f: stopping tablet replica
I20260621 05:24:52.865435 20252 raft_consensus.cc:2258] T 98236352a4954ac1a96e1367425e9368 P e6f1c8b21fec4ce5ac34e7cd6a4fe6d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:24:52.865511 20254 raft_consensus.cc:2229] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:24:52.865600 20254 raft_consensus.cc:2258] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:24:52.865751 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 19825
I20260621 05:24:52.865850 20252 ts_tablet_manager.cc:1905] T 98236352a4954ac1a96e1367425e9368 P e6f1c8b21fec4ce5ac34e7cd6a4fe6d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:24:52.866045 20253 tablet_replica.cc:330] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438: stopping tablet replica
I20260621 05:24:52.866245 20253 raft_consensus.cc:2229] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:24:52.866370 20253 raft_consensus.cc:2258] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:24:52.867013 20253 ts_tablet_manager.cc:1905] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:24:52.868127 20254 ts_tablet_manager.cc:1905] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:24:52.868896 20253 ts_tablet_manager.cc:1918] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:24:52.868959 20253 log.cc:1196] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/98236352a4954ac1a96e1367425e9368
I20260621 05:24:52.869199 20253 ts_tablet_manager.cc:1939] T 98236352a4954ac1a96e1367425e9368 P 496f53d6fe1242efb3aa8201df828438: Deleting consensus metadata
I20260621 05:24:52.869361 20254 ts_tablet_manager.cc:1918] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:24:52.869407 20254 log.cc:1196] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/98236352a4954ac1a96e1367425e9368
I20260621 05:24:52.869589 19752 catalog_manager.cc:4841] TS 496f53d6fe1242efb3aa8201df828438 (127.2.32.2:34859): tablet 98236352a4954ac1a96e1367425e9368 (table test-table [id=c629e15e0c8540be99225482e67ab29b]) successfully deleted
I20260621 05:24:52.869663 20254 ts_tablet_manager.cc:1939] T 98236352a4954ac1a96e1367425e9368 P 0dbccd4274b04fb9b5d9f125d919640f: Deleting consensus metadata
I20260621 05:24:52.870030 19749 catalog_manager.cc:4841] TS 0dbccd4274b04fb9b5d9f125d919640f (127.2.32.3:41319): tablet 98236352a4954ac1a96e1367425e9368 (table test-table [id=c629e15e0c8540be99225482e67ab29b]) successfully deleted
W20260621 05:24:52.871454 19751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.32.1:34873 (error 108)
I20260621 05:24:52.871677 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 19956
I20260621 05:24:52.872282 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:52.871764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:34873 (local address 127.0.0.1:40974)
0621 05:24:52.872200 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:52.872244 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.1:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415}
W20260621 05:24:52.872393 19751 catalog_manager.cc:4584] TS e6f1c8b21fec4ce5ac34e7cd6a4fe6d5 (127.2.32.1:34873): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98236352a4954ac1a96e1367425e9368: Network error: Client connection negotiation failed: client connection to 127.2.32.1:34873: connect: Connection refused (error 111)
I20260621 05:24:52.876204 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 20087
I20260621 05:24:52.883263 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 19734
2026-06-21T05:24:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (892 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:24:52 dist-test-slave-b357 krb5kdc[20258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:24:52 dist-test-slave-b357 krb5kdc[20258](info): set up 2 sockets
Jun 21 05:24:52 dist-test-slave-b357 krb5kdc[20258](info): commencing operation
krb5kdc: starting...
W20260621 05:24:53.971414 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:24:54 dist-test-slave-b357 krb5kdc[20258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:24:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:24:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:24:54.101816 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:43559
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:43559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:24:54.184711 20271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:54.184960 20271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:54.185002 20271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:54.186450 20271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:24:54.186510 20271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:54.186549 20271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:54.186582 20271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:24:54.186610 20271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:24:54.188143 20271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:43559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:43559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:54.188406 20271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:54.188621 20271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:54.193897 20276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:54.193928 20279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:54.193986 20277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:54.194257 20271 server_base.cc:1034] running on GCE node
I20260621 05:24:54.194636 20271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:54.194979 20271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:54.196166 20271 hybrid_clock.cc:648] HybridClock initialized: now 1782019494196138 us; error 44 us; skew 500 ppm
Jun 21 05:24:54 dist-test-slave-b357 krb5kdc[20258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019494, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:54.198855 20271 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:24:54.199501 20271 webserver.cc:460] Webserver started at http://127.2.32.62:46697/ using document root <none> and password file <none>
I20260621 05:24:54.199729 20271 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:54.199795 20271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:54.199942 20271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:54.200961 20271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "08a71040164a44749ca9f9325309354e"
format_stamp: "Formatted at 2026-06-21 05:24:54 on dist-test-slave-b357"
server_key: "55ce64f0208282a8a6f65375f8848e09"
server_key_iv: "20c0a1bfd9880304921b53ee326045de"
server_key_version: "encryptionkey@0"
I20260621 05:24:54.201315 20271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "08a71040164a44749ca9f9325309354e"
format_stamp: "Formatted at 2026-06-21 05:24:54 on dist-test-slave-b357"
server_key: "55ce64f0208282a8a6f65375f8848e09"
server_key_iv: "20c0a1bfd9880304921b53ee326045de"
server_key_version: "encryptionkey@0"
I20260621 05:24:54.202607 20271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:24:54.203445 20286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:54.203614 20271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:24:54.203713 20271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "08a71040164a44749ca9f9325309354e"
format_stamp: "Formatted at 2026-06-21 05:24:54 on dist-test-slave-b357"
server_key: "55ce64f0208282a8a6f65375f8848e09"
server_key_iv: "20c0a1bfd9880304921b53ee326045de"
server_key_version: "encryptionkey@0"
I20260621 05:24:54.203791 20271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:54.220083 20271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:54.222838 20271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:54.223201 20271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:54.228744 20271 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:43559
I20260621 05:24:54.228785 20338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:43559 every 8 connection(s)
I20260621 05:24:54.229206 20271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:24:54.230116 20339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:54.232434 20339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e: Bootstrap starting.
I20260621 05:24:54.232997 20339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:54.233275 20339 log.cc:824] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:54.234337 20339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e: No bootstrap required, opened a new log
I20260621 05:24:54.236285 20339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a71040164a44749ca9f9325309354e" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43559 } }
I20260621 05:24:54.236455 20339 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:54.236480 20339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a71040164a44749ca9f9325309354e, State: Initialized, Role: FOLLOWER
I20260621 05:24:54.236687 20339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a71040164a44749ca9f9325309354e" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43559 } }
I20260621 05:24:54.236784 20339 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:24:54.236825 20339 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:24:54.236881 20339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:54.236881 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 20271
I20260621 05:24:54.237023 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:24:54.237250 2176 external_mini_cluster.cc:1428] Setting key 7fe44eda0aa8a8828cdc795fd2aea423
I20260621 05:24:54.237848 20339 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a71040164a44749ca9f9325309354e" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43559 } }
I20260621 05:24:54.237975 20339 leader_election.cc:302] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08a71040164a44749ca9f9325309354e; no voters:
I20260621 05:24:54.238230 20339 leader_election.cc:288] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:24:54.238298 20344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:24:54.238473 20344 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e [term 1 LEADER]: Becoming Leader. State: Replica: 08a71040164a44749ca9f9325309354e, State: Running, Role: LEADER
I20260621 05:24:54.238533 20339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:24:54.238579 20344 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a71040164a44749ca9f9325309354e" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43559 } }
I20260621 05:24:54.238955 20344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08a71040164a44749ca9f9325309354e. Latest consensus state: current_term: 1 leader_uuid: "08a71040164a44749ca9f9325309354e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a71040164a44749ca9f9325309354e" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43559 } } }
I20260621 05:24:54.239041 20344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e [sys.catalog]: This master's current role is: LEADER
I20260621 05:24:54.239218 20346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08a71040164a44749ca9f9325309354e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a71040164a44749ca9f9325309354e" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43559 } } }
I20260621 05:24:54.239296 20349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:24:54.239368 20346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e [sys.catalog]: This master's current role is: LEADER
I20260621 05:24:54.239873 20349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:24:54.241626 20349 catalog_manager.cc:1347] Generated new cluster ID: d253c9c186954b28876934f0162f5b0c
I20260621 05:24:54.241679 20349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:24:54 dist-test-slave-b357 krb5kdc[20258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:24:54.248124 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:54.237852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35024 (local address 127.2.32.62:43559)
0621 05:24:54.238177 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:54.238190 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:54.238233 (+ 43us) server_negotiation.cc:407] Connection header received
0621 05:24:54.238391 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:54.238399 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:54.238464 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:54.239695 (+ 1231us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:54.239809 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.241275 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.242151 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.242559 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.242671 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:54.244884 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:54.244888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:54.244889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:54.244899 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:54.246514 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:54.246922 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:54.246925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:54.246926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:54.247038 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:54.247268 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:54.247269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:54.247269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:54.247506 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:54.247668 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:54.247742 (+ 74us) server_negotiation.cc:299] Negotiation successful
0621 05:24:54.247976 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":126,"threads_started":1}
I20260621 05:24:54.262066 20349 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:24:54.262735 20349 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:24:54.271410 20349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e: Generated new TSK 0
I20260621 05:24:54.271716 20349 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
I20260621 05:24:54.304214 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:43559
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:24:54.389693 20365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:54.389915 20365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:54.389940 20365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:54.391048 20365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:54.391091 20365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:54.391119 20365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:24:54.392805 20365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:43559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:54.393082 20365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:54.393318 20365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:54.398201 20371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:54.398197 20370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:54.398504 20373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:54.398590 20365 server_base.cc:1034] running on GCE node
I20260621 05:24:54.398811 20365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:54.399075 20365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:54.400270 20365 hybrid_clock.cc:648] HybridClock initialized: now 1782019494400259 us; error 32 us; skew 500 ppm
Jun 21 05:24:54 dist-test-slave-b357 krb5kdc[20258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019494, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:54.403189 20365 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:24:54.403782 20365 webserver.cc:460] Webserver started at http://127.2.32.1:44001/ using document root <none> and password file <none>
I20260621 05:24:54.404053 20365 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:54.404130 20365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:54.404287 20365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:54.405526 20365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "801385d835164ad1b65f5ebdb111f49b"
format_stamp: "Formatted at 2026-06-21 05:24:54 on dist-test-slave-b357"
server_key: "6ed8a5e5185838eedb8e5e8a6ed7f4ca"
server_key_iv: "5d017fb2af0e853a6732207e881cf637"
server_key_version: "encryptionkey@0"
I20260621 05:24:54.405946 20365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "801385d835164ad1b65f5ebdb111f49b"
format_stamp: "Formatted at 2026-06-21 05:24:54 on dist-test-slave-b357"
server_key: "6ed8a5e5185838eedb8e5e8a6ed7f4ca"
server_key_iv: "5d017fb2af0e853a6732207e881cf637"
server_key_version: "encryptionkey@0"
I20260621 05:24:54.407666 20365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260621 05:24:54.408674 20380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:54.408965 20365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:54.409070 20365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "801385d835164ad1b65f5ebdb111f49b"
format_stamp: "Formatted at 2026-06-21 05:24:54 on dist-test-slave-b357"
server_key: "6ed8a5e5185838eedb8e5e8a6ed7f4ca"
server_key_iv: "5d017fb2af0e853a6732207e881cf637"
server_key_version: "encryptionkey@0"
I20260621 05:24:54.409144 20365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:54.426393 20365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:54.428589 20365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:54.428699 20365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:54.428881 20365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:54.429209 20365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:54.429248 20365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:54.429270 20365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:54.429283 20365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:54.436429 20365 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:36019
I20260621 05:24:54.436513 20493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:36019 every 8 connection(s)
I20260621 05:24:54.436899 20365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:24:54.439318 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 20365
I20260621 05:24:54.439422 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:24:54.439635 2176 external_mini_cluster.cc:1428] Setting key 44f28fcf327212c4f1a474a044fddee0
Jun 21 05:24:54 dist-test-slave-b357 krb5kdc[20258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019494, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:24:54.446815 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:54.437976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:42791 (local address 127.2.32.62:43559)
0621 05:24:54.438029 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:54.438032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:54.438354 (+ 322us) server_negotiation.cc:407] Connection header received
0621 05:24:54.438928 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:54.438930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:54.438943 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:54.439002 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:54.440219 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.440748 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.441779 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.441992 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.442076 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:54.444130 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:54.444132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:54.444133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:54.444140 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:54.445487 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:54.445889 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:54.445890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:54.445890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:54.445944 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:54.446179 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:54.446180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:54.446180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:54.446291 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:54.446359 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:54.446642 (+ 283us) server_negotiation.cc:299] Negotiation successful
0621 05:24:54.446728 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
WARNING: no policy specified for kudu/127.2.32.2@KRBTEST.COM; defaulting to no policy
I20260621 05:24:54.446995 20496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:54.437856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:43559 (local address 127.2.32.1:42791)
0621 05:24:54.438212 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:54.438255 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:54.438804 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:54.439085 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:54.439088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:54.439886 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:54.440082 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:54.440087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.440868 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.440869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:54.441679 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:54.441681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.442085 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.442087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:54.442213 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:54.442700 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:24:54.442708 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:54.444039 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:54.445585 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:54.445592 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:54.445594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:54.445802 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:54.446006 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:54.446006 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:54.446007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:54.446112 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:54.446373 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:54.446379 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:54.446504 (+ 125us) client_negotiation.cc:769] Sending connection context
0621 05:24:54.446619 (+ 115us) client_negotiation.cc:240] Negotiation successful
0621 05:24:54.446767 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":222,"threads_started":1}
I20260621 05:24:54.447695 20494 heartbeater.cc:347] Connected to a master server at 127.2.32.62:43559
I20260621 05:24:54.447781 20494 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:54.447956 20494 heartbeater.cc:511] Master 127.2.32.62:43559 requested a full tablet report, sending...
I20260621 05:24:54.448580 20303 ts_manager.cc:194] Registered new tserver with Master: 801385d835164ad1b65f5ebdb111f49b (127.2.32.1:36019)
I20260621 05:24:54.449172 20303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:42791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
I20260621 05:24:54.466634 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:43559
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:24:54.554123 20499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:54.554312 20499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:54.554334 20499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:54.555431 20499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:54.555469 20499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:54.555493 20499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:24:54.556833 20499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:43559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:54.557001 20499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:54.557160 20499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:54.562163 20504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:54.562175 20507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:54.562225 20505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:54.562363 20499 server_base.cc:1034] running on GCE node
I20260621 05:24:54.562649 20499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:54.562863 20499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:54.564077 20499 hybrid_clock.cc:648] HybridClock initialized: now 1782019494564048 us; error 31 us; skew 500 ppm
Jun 21 05:24:54 dist-test-slave-b357 krb5kdc[20258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019494, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:54.567533 20499 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:24:54.568171 20499 webserver.cc:460] Webserver started at http://127.2.32.2:41925/ using document root <none> and password file <none>
I20260621 05:24:54.568392 20499 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:54.568467 20499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:54.568630 20499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:54.569700 20499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "293051fc831e494b83aaec34a6a9f8bd"
format_stamp: "Formatted at 2026-06-21 05:24:54 on dist-test-slave-b357"
server_key: "ba28b1b3dbeb88c82b985f24aae4816c"
server_key_iv: "15d71f8aab707e3f272503a7176f0760"
server_key_version: "encryptionkey@0"
I20260621 05:24:54.570057 20499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "293051fc831e494b83aaec34a6a9f8bd"
format_stamp: "Formatted at 2026-06-21 05:24:54 on dist-test-slave-b357"
server_key: "ba28b1b3dbeb88c82b985f24aae4816c"
server_key_iv: "15d71f8aab707e3f272503a7176f0760"
server_key_version: "encryptionkey@0"
I20260621 05:24:54.571617 20499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:24:54.572686 20514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:54.572909 20499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:54.573015 20499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "293051fc831e494b83aaec34a6a9f8bd"
format_stamp: "Formatted at 2026-06-21 05:24:54 on dist-test-slave-b357"
server_key: "ba28b1b3dbeb88c82b985f24aae4816c"
server_key_iv: "15d71f8aab707e3f272503a7176f0760"
server_key_version: "encryptionkey@0"
I20260621 05:24:54.573091 20499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:54.598821 20499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:54.601281 20499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:54.601447 20499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:54.601691 20499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:54.601994 20499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:54.602046 20499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:54.602085 20499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:54.602116 20499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:54.608328 20499 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:45357
I20260621 05:24:54.608376 20627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:45357 every 8 connection(s)
I20260621 05:24:54.608676 20499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:24:54.612632 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 20499
I20260621 05:24:54.612784 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:24:54.613024 2176 external_mini_cluster.cc:1428] Setting key 90029b99f1c1a2e201b2750e80ceab46
Jun 21 05:24:54 dist-test-slave-b357 krb5kdc[20258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019494, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:24:54.619864 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:54.609468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:42125 (local address 127.2.32.62:43559)
0621 05:24:54.609550 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:54.609552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:54.609724 (+ 172us) server_negotiation.cc:407] Connection header received
0621 05:24:54.610251 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:54.610252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:54.610263 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:54.610322 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:54.611288 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.611885 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.613407 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.613622 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.613700 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:54.616682 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:54.616684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:54.616684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:54.616692 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:54.618435 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:54.618871 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:54.618873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:54.618873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:54.618933 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:54.619229 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:54.619230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:54.619230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:54.619347 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:54.619419 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:54.619691 (+ 272us) server_negotiation.cc:299] Negotiation successful
0621 05:24:54.619771 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 05:24:54.620070 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:54.609444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:43559 (local address 127.2.32.2:42125)
0621 05:24:54.609610 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:54.609643 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:54.610137 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:54.610369 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:54.610372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:54.610898 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:54.611079 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:54.611083 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.612001 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.612002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:54.613296 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:54.613301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.613706 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.613708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:54.613843 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:54.614398 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:24:54.614416 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:54.616589 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:54.618540 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:54.618546 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:54.618549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:54.618786 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:54.619011 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:54.619011 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:54.619012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:54.619160 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:54.619411 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:54.619418 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:54.619558 (+ 140us) client_negotiation.cc:769] Sending connection context
0621 05:24:54.619658 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 05:24:54.619834 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":111,"threads_started":1}
I20260621 05:24:54.620535 20628 heartbeater.cc:347] Connected to a master server at 127.2.32.62:43559
I20260621 05:24:54.620625 20628 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:54.620810 20628 heartbeater.cc:511] Master 127.2.32.62:43559 requested a full tablet report, sending...
I20260621 05:24:54.621340 20303 ts_manager.cc:194] Registered new tserver with Master: 293051fc831e494b83aaec34a6a9f8bd (127.2.32.2:45357)
I20260621 05:24:54.622021 20303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:42125
WARNING: no policy specified for kudu/127.2.32.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
I20260621 05:24:54.640599 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:43559
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:24:54.723268 20633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:54.723460 20633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:54.723483 20633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:54.724658 20633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:54.724699 20633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:54.724727 20633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:24:54.726125 20633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:43559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:54.726341 20633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:54.726586 20633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:54.732017 20638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:54.732016 20639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:54.732182 20641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:54.732508 20633 server_base.cc:1034] running on GCE node
I20260621 05:24:54.732779 20633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:54.733037 20633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:54.734220 20633 hybrid_clock.cc:648] HybridClock initialized: now 1782019494734214 us; error 36 us; skew 500 ppm
Jun 21 05:24:54 dist-test-slave-b357 krb5kdc[20258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019494, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:54.738180 20633 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:24:54.738722 20633 webserver.cc:460] Webserver started at http://127.2.32.3:34539/ using document root <none> and password file <none>
I20260621 05:24:54.738968 20633 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:54.739034 20633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:54.739193 20633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:54.740439 20633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "d6727e1ba04443b582dd7e3c1ebdae35"
format_stamp: "Formatted at 2026-06-21 05:24:54 on dist-test-slave-b357"
server_key: "6555e1e7cf29ea59dd4900329becef43"
server_key_iv: "0e112d5a40695c7848b31536b71f49f1"
server_key_version: "encryptionkey@0"
I20260621 05:24:54.740854 20633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "d6727e1ba04443b582dd7e3c1ebdae35"
format_stamp: "Formatted at 2026-06-21 05:24:54 on dist-test-slave-b357"
server_key: "6555e1e7cf29ea59dd4900329becef43"
server_key_iv: "0e112d5a40695c7848b31536b71f49f1"
server_key_version: "encryptionkey@0"
I20260621 05:24:54.742136 20633 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:24:54.742913 20648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:54.743251 20633 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:54.743378 20633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "d6727e1ba04443b582dd7e3c1ebdae35"
format_stamp: "Formatted at 2026-06-21 05:24:54 on dist-test-slave-b357"
server_key: "6555e1e7cf29ea59dd4900329becef43"
server_key_iv: "0e112d5a40695c7848b31536b71f49f1"
server_key_version: "encryptionkey@0"
I20260621 05:24:54.743471 20633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:54.763960 20633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:54.766199 20633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:54.766355 20633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:54.766605 20633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:54.766919 20633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:54.766981 20633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:54.767045 20633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:54.767082 20633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:54.774012 20633 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:34643
I20260621 05:24:54.774137 20761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:34643 every 8 connection(s)
I20260621 05:24:54.774416 20633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:24:54.775946 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 20633
I20260621 05:24:54.776048 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:24:54.776265 2176 external_mini_cluster.cc:1428] Setting key 4f7fcbcde503c073f7632a18b1c6c569
Jun 21 05:24:54 dist-test-slave-b357 krb5kdc[20258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019494, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:24:54.785199 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:54.775199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:51257 (local address 127.2.32.62:43559)
0621 05:24:54.775251 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:54.775253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:54.775458 (+ 205us) server_negotiation.cc:407] Connection header received
0621 05:24:54.776189 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:54.776192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:54.776215 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:54.776310 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:54.777417 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.777994 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.779202 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.779436 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.779517 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:54.781915 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:54.781917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:54.781917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:54.781928 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:54.783713 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:54.784249 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:54.784250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:54.784251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:54.784332 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:54.784604 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:54.784605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:54.784606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:54.784744 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:54.784811 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:54.785062 (+ 251us) server_negotiation.cc:299] Negotiation successful
0621 05:24:54.785133 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:24:54.785346 20764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:54.775171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:43559 (local address 127.2.32.3:51257)
0621 05:24:54.775340 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:54.775375 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:54.776033 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:54.776409 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:54.776413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:54.777093 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:54.777311 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:54.777315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.778111 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.778113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:54.779033 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:54.779036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.779539 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.779540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:54.779643 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:54.780114 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:24:54.780122 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:54.781825 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:54.783899 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:54.783906 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:54.783907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:54.784137 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:54.784400 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:54.784401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:54.784401 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:54.784508 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:54.784813 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:54.784818 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:54.784932 (+ 114us) client_negotiation.cc:769] Sending connection context
0621 05:24:54.785024 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 05:24:54.785152 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260621 05:24:54.785815 20762 heartbeater.cc:347] Connected to a master server at 127.2.32.62:43559
I20260621 05:24:54.785925 20762 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:54.786111 20762 heartbeater.cc:511] Master 127.2.32.62:43559 requested a full tablet report, sending...
I20260621 05:24:54.786599 20303 ts_manager.cc:194] Registered new tserver with Master: d6727e1ba04443b582dd7e3c1ebdae35 (127.2.32.3:34643)
I20260621 05:24:54.786978 20303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:51257
I20260621 05:24:54.788296 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:24:54.796877 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:54.789598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35036 (local address 127.2.32.62:43559)
0621 05:24:54.789704 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:54.789706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:54.789826 (+ 120us) server_negotiation.cc:407] Connection header received
0621 05:24:54.789903 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:54.789904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:54.789915 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:54.789980 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:54.790533 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.791026 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.791753 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.791946 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.792017 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:54.793739 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:54.793741 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:54.793741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:54.793752 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:54.795439 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:54.796034 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:54.796036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:54.796036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:54.796097 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:54.796372 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:54.796373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:54.796374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:54.796520 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:54.796605 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:54.796764 (+ 159us) server_negotiation.cc:299] Negotiation successful
0621 05:24:54.796810 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 05:24:54.798731 20303 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:24:54.800822 20303 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:24:54.808352 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:54.803608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:36019 (local address 127.0.0.1:38274)
0621 05:24:54.803837 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:54.803894 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:54.803972 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:54.804273 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:54.804275 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:54.804277 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:54.804400 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:54.804403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.805689 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.805690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:54.806973 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:54.806976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.808043 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.808044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:54.808126 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:54.808141 (+ 15us) client_negotiation.cc:769] Sending connection context
0621 05:24:54.808208 (+ 67us) client_negotiation.cc:240] Negotiation successful
0621 05:24:54.808268 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":64,"threads_started":1}
I20260621 05:24:54.808626 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:54.803667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:45357 (local address 127.0.0.1:43248)
0621 05:24:54.804055 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:54.804069 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:54.804503 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:54.804821 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:54.804822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:54.804825 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:54.804898 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:54.804900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.805886 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.805887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:54.807231 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:54.807234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.808279 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.808280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:54.808365 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:54.808371 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 05:24:54.808439 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 05:24:54.808478 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":3,"thread_start_us":110,"threads_started":1}
I20260621 05:24:54.808626 20777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:54.803897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:34643 (local address 127.0.0.1:55342)
0621 05:24:54.804113 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:54.804124 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:54.804175 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:54.804898 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:54.804899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:54.804902 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:54.804948 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:54.804950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.806214 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.806215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:54.807454 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:54.807456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.808281 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.808282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:54.808365 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:54.808374 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 05:24:54.808407 (+ 33us) client_negotiation.cc:240] Negotiation successful
0621 05:24:54.808478 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":4,"thread_start_us":82,"threads_started":1}
I20260621 05:24:54.808996 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:54.804227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55342 (local address 127.2.32.3:34643)
0621 05:24:54.804726 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:54.804730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:54.804741 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:24:54.804778 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:54.804780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:54.804795 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:54.805011 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:54.805083 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.806105 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.807546 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.808157 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.808667 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:54.808833 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:54.808866 (+ 33us) server_negotiation.cc:299] Negotiation successful
0621 05:24:54.808924 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":126,"threads_started":1}
I20260621 05:24:54.809013 20775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:54.803715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.2.32.1:36019)
0621 05:24:54.803896 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:54.803900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:54.804060 (+ 160us) server_negotiation.cc:407] Connection header received
0621 05:24:54.804105 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:54.804107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:54.804124 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:54.804196 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:54.804484 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.805582 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.807051 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.807884 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.807952 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:54.808007 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:54.808891 (+ 884us) server_negotiation.cc:299] Negotiation successful
0621 05:24:54.808933 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":84,"threads_started":1}
I20260621 05:24:54.809108 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:54.803814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43248 (local address 127.2.32.2:45357)
0621 05:24:54.804043 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:54.804048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:54.804461 (+ 413us) server_negotiation.cc:407] Connection header received
0621 05:24:54.804699 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:54.804702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:54.804721 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:54.804827 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:54.804961 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.805772 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.807389 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.808151 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.808233 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:54.808283 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:54.808700 (+ 417us) server_negotiation.cc:299] Negotiation successful
0621 05:24:54.809013 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":115,"threads_started":1}
I20260621 05:24:54.809684 20562 tablet_service.cc:1467] Processing CreateTablet for tablet ee65243e7c274a2f9f281581dfadb6a2 (DEFAULT_TABLE table=test-table [id=b8eb4fb1a9314f97bcdd0d983c06c70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:54.809978 20562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee65243e7c274a2f9f281581dfadb6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:54.809870 20428 tablet_service.cc:1467] Processing CreateTablet for tablet ee65243e7c274a2f9f281581dfadb6a2 (DEFAULT_TABLE table=test-table [id=b8eb4fb1a9314f97bcdd0d983c06c70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:54.810312 20428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee65243e7c274a2f9f281581dfadb6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:54.812686 20779 tablet_bootstrap.cc:492] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd: Bootstrap starting.
I20260621 05:24:54.813352 20779 tablet_bootstrap.cc:654] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:54.813853 20779 log.cc:824] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:54.813937 20780 tablet_bootstrap.cc:492] T ee65243e7c274a2f9f281581dfadb6a2 P 801385d835164ad1b65f5ebdb111f49b: Bootstrap starting.
I20260621 05:24:54.814587 20780 tablet_bootstrap.cc:654] T ee65243e7c274a2f9f281581dfadb6a2 P 801385d835164ad1b65f5ebdb111f49b: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:54.814499 20696 tablet_service.cc:1467] Processing CreateTablet for tablet ee65243e7c274a2f9f281581dfadb6a2 (DEFAULT_TABLE table=test-table [id=b8eb4fb1a9314f97bcdd0d983c06c70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:54.814616 20779 tablet_bootstrap.cc:492] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd: No bootstrap required, opened a new log
I20260621 05:24:54.814709 20779 ts_tablet_manager.cc:1397] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:24:54.814865 20696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee65243e7c274a2f9f281581dfadb6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:54.815182 20780 log.cc:824] T ee65243e7c274a2f9f281581dfadb6a2 P 801385d835164ad1b65f5ebdb111f49b: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:54.815943 20780 tablet_bootstrap.cc:492] T ee65243e7c274a2f9f281581dfadb6a2 P 801385d835164ad1b65f5ebdb111f49b: No bootstrap required, opened a new log
I20260621 05:24:54.816016 20780 ts_tablet_manager.cc:1397] T ee65243e7c274a2f9f281581dfadb6a2 P 801385d835164ad1b65f5ebdb111f49b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:24:54.816576 20779 raft_consensus.cc:348] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801385d835164ad1b65f5ebdb111f49b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36019 } } peers { permanent_uuid: "293051fc831e494b83aaec34a6a9f8bd" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45357 } } peers { permanent_uuid: "d6727e1ba04443b582dd7e3c1ebdae35" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34643 } }
I20260621 05:24:54.816730 20779 raft_consensus.cc:374] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:54.816772 20779 raft_consensus.cc:729] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 293051fc831e494b83aaec34a6a9f8bd, State: Initialized, Role: FOLLOWER
I20260621 05:24:54.816879 20779 consensus_queue.cc:260] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801385d835164ad1b65f5ebdb111f49b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36019 } } peers { permanent_uuid: "293051fc831e494b83aaec34a6a9f8bd" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45357 } } peers { permanent_uuid: "d6727e1ba04443b582dd7e3c1ebdae35" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34643 } }
I20260621 05:24:54.817148 20779 ts_tablet_manager.cc:1428] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:24:54.817273 20628 heartbeater.cc:503] Master 127.2.32.62:43559 was elected leader, sending a full tablet report...
I20260621 05:24:54.818152 20780 raft_consensus.cc:348] T ee65243e7c274a2f9f281581dfadb6a2 P 801385d835164ad1b65f5ebdb111f49b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801385d835164ad1b65f5ebdb111f49b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36019 } } peers { permanent_uuid: "293051fc831e494b83aaec34a6a9f8bd" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45357 } } peers { permanent_uuid: "d6727e1ba04443b582dd7e3c1ebdae35" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34643 } }
I20260621 05:24:54.818328 20780 raft_consensus.cc:374] T ee65243e7c274a2f9f281581dfadb6a2 P 801385d835164ad1b65f5ebdb111f49b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:54.818367 20780 raft_consensus.cc:729] T ee65243e7c274a2f9f281581dfadb6a2 P 801385d835164ad1b65f5ebdb111f49b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 801385d835164ad1b65f5ebdb111f49b, State: Initialized, Role: FOLLOWER
I20260621 05:24:54.818468 20780 consensus_queue.cc:260] T ee65243e7c274a2f9f281581dfadb6a2 P 801385d835164ad1b65f5ebdb111f49b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801385d835164ad1b65f5ebdb111f49b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36019 } } peers { permanent_uuid: "293051fc831e494b83aaec34a6a9f8bd" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45357 } } peers { permanent_uuid: "d6727e1ba04443b582dd7e3c1ebdae35" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34643 } }
I20260621 05:24:54.818712 20780 ts_tablet_manager.cc:1428] T ee65243e7c274a2f9f281581dfadb6a2 P 801385d835164ad1b65f5ebdb111f49b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 05:24:54.818846 20494 heartbeater.cc:503] Master 127.2.32.62:43559 was elected leader, sending a full tablet report...
I20260621 05:24:54.818835 20784 tablet_bootstrap.cc:492] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35: Bootstrap starting.
I20260621 05:24:54.819832 20784 tablet_bootstrap.cc:654] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:54.820345 20784 log.cc:824] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:54.821041 20784 tablet_bootstrap.cc:492] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35: No bootstrap required, opened a new log
I20260621 05:24:54.821138 20784 ts_tablet_manager.cc:1397] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:24:54.822639 20784 raft_consensus.cc:348] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801385d835164ad1b65f5ebdb111f49b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36019 } } peers { permanent_uuid: "293051fc831e494b83aaec34a6a9f8bd" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45357 } } peers { permanent_uuid: "d6727e1ba04443b582dd7e3c1ebdae35" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34643 } }
I20260621 05:24:54.822781 20784 raft_consensus.cc:374] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:54.822825 20784 raft_consensus.cc:729] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6727e1ba04443b582dd7e3c1ebdae35, State: Initialized, Role: FOLLOWER
I20260621 05:24:54.822952 20784 consensus_queue.cc:260] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801385d835164ad1b65f5ebdb111f49b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36019 } } peers { permanent_uuid: "293051fc831e494b83aaec34a6a9f8bd" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45357 } } peers { permanent_uuid: "d6727e1ba04443b582dd7e3c1ebdae35" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34643 } }
I20260621 05:24:54.823249 20784 ts_tablet_manager.cc:1428] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:24:54.823307 20762 heartbeater.cc:503] Master 127.2.32.62:43559 was elected leader, sending a full tablet report...
W20260621 05:24:54.859401 20629 tablet.cc:2369] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:24:54.891268 20783 raft_consensus.cc:482] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:24:54.891446 20783 raft_consensus.cc:504] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801385d835164ad1b65f5ebdb111f49b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36019 } } peers { permanent_uuid: "293051fc831e494b83aaec34a6a9f8bd" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45357 } } peers { permanent_uuid: "d6727e1ba04443b582dd7e3c1ebdae35" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34643 } }
I20260621 05:24:54.891918 20783 leader_election.cc:288] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 801385d835164ad1b65f5ebdb111f49b (127.2.32.1:36019), d6727e1ba04443b582dd7e3c1ebdae35 (127.2.32.3:34643)
I20260621 05:24:54.894980 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:54.892290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:55143 (local address 127.2.32.3:34643)
0621 05:24:54.892344 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:54.892346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:54.892356 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:24:54.892393 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:54.892394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:54.892407 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:54.892483 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:54.892725 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.893256 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.894213 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.894610 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.894690 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:54.894703 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:54.894854 (+ 151us) server_negotiation.cc:299] Negotiation successful
0621 05:24:54.894900 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260621 05:24:54.894980 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:54.892173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:34643 (local address 127.2.32.2:55143)
0621 05:24:54.892252 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:54.892267 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:54.892326 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:54.892486 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:54.892488 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:54.892490 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:54.892588 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:54.892590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.893338 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.893339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:54.894122 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:54.894124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.894699 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.894700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:54.894783 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:54.894786 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:24:54.894849 (+ 63us) client_negotiation.cc:240] Negotiation successful
0621 05:24:54.894897 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 05:24:54.895481 20775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:54.892261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:58971 (local address 127.2.32.1:36019)
0621 05:24:54.892322 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:54.892324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:54.892497 (+ 173us) server_negotiation.cc:407] Connection header received
0621 05:24:54.892655 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:54.892656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:54.892671 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:54.892742 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:54.892901 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.893506 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.894589 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.894981 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.895062 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:54.895075 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:54.895371 (+ 296us) server_negotiation.cc:299] Negotiation successful
0621 05:24:54.895426 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260621 05:24:54.895496 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:54.892175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:36019 (local address 127.2.32.2:58971)
0621 05:24:54.892414 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:54.892423 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:54.892569 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:54.892746 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:54.892747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:54.892749 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:54.892833 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:54.892834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.893656 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.893658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:54.894489 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:54.894491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.895135 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.895136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:54.895252 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:54.895257 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:24:54.895342 (+ 85us) client_negotiation.cc:240] Negotiation successful
0621 05:24:54.895396 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":15,"thread_start_us":146,"threads_started":1}
I20260621 05:24:54.895481 20716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee65243e7c274a2f9f281581dfadb6a2" candidate_uuid: "293051fc831e494b83aaec34a6a9f8bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6727e1ba04443b582dd7e3c1ebdae35" is_pre_election: true
I20260621 05:24:54.895682 20716 raft_consensus.cc:2454] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 293051fc831e494b83aaec34a6a9f8bd in term 0.
I20260621 05:24:54.895892 20448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee65243e7c274a2f9f281581dfadb6a2" candidate_uuid: "293051fc831e494b83aaec34a6a9f8bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "801385d835164ad1b65f5ebdb111f49b" is_pre_election: true
I20260621 05:24:54.895975 20517 leader_election.cc:302] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 293051fc831e494b83aaec34a6a9f8bd, d6727e1ba04443b582dd7e3c1ebdae35; no voters:
I20260621 05:24:54.896044 20448 raft_consensus.cc:2454] T ee65243e7c274a2f9f281581dfadb6a2 P 801385d835164ad1b65f5ebdb111f49b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 293051fc831e494b83aaec34a6a9f8bd in term 0.
I20260621 05:24:54.896191 20783 raft_consensus.cc:2781] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:24:54.896311 20783 raft_consensus.cc:482] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:24:54.896344 20783 raft_consensus.cc:3037] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:54.897219 20783 raft_consensus.cc:504] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801385d835164ad1b65f5ebdb111f49b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36019 } } peers { permanent_uuid: "293051fc831e494b83aaec34a6a9f8bd" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45357 } } peers { permanent_uuid: "d6727e1ba04443b582dd7e3c1ebdae35" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34643 } }
I20260621 05:24:54.897418 20783 leader_election.cc:288] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [CANDIDATE]: Term 1 election: Requested vote from peers 801385d835164ad1b65f5ebdb111f49b (127.2.32.1:36019), d6727e1ba04443b582dd7e3c1ebdae35 (127.2.32.3:34643)
I20260621 05:24:54.897620 20448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee65243e7c274a2f9f281581dfadb6a2" candidate_uuid: "293051fc831e494b83aaec34a6a9f8bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "801385d835164ad1b65f5ebdb111f49b"
I20260621 05:24:54.897614 20716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee65243e7c274a2f9f281581dfadb6a2" candidate_uuid: "293051fc831e494b83aaec34a6a9f8bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6727e1ba04443b582dd7e3c1ebdae35"
I20260621 05:24:54.897706 20716 raft_consensus.cc:3037] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:54.897706 20448 raft_consensus.cc:3037] T ee65243e7c274a2f9f281581dfadb6a2 P 801385d835164ad1b65f5ebdb111f49b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:54.898676 20716 raft_consensus.cc:2454] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 293051fc831e494b83aaec34a6a9f8bd in term 1.
I20260621 05:24:54.898676 20448 raft_consensus.cc:2454] T ee65243e7c274a2f9f281581dfadb6a2 P 801385d835164ad1b65f5ebdb111f49b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 293051fc831e494b83aaec34a6a9f8bd in term 1.
I20260621 05:24:54.898939 20515 leader_election.cc:302] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 293051fc831e494b83aaec34a6a9f8bd, 801385d835164ad1b65f5ebdb111f49b; no voters:
I20260621 05:24:54.899077 20783 raft_consensus.cc:2781] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:24:54.899286 20783 raft_consensus.cc:686] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [term 1 LEADER]: Becoming Leader. State: Replica: 293051fc831e494b83aaec34a6a9f8bd, State: Running, Role: LEADER
I20260621 05:24:54.899389 20783 consensus_queue.cc:237] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801385d835164ad1b65f5ebdb111f49b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36019 } } peers { permanent_uuid: "293051fc831e494b83aaec34a6a9f8bd" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45357 } } peers { permanent_uuid: "d6727e1ba04443b582dd7e3c1ebdae35" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34643 } }
I20260621 05:24:54.900229 20303 catalog_manager.cc:5495] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 293051fc831e494b83aaec34a6a9f8bd (127.2.32.2). New cstate: current_term: 1 leader_uuid: "293051fc831e494b83aaec34a6a9f8bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801385d835164ad1b65f5ebdb111f49b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "293051fc831e494b83aaec34a6a9f8bd" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 45357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6727e1ba04443b582dd7e3c1ebdae35" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34643 } health_report { overall_health: UNKNOWN } } }
I20260621 05:24:54.928682 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:54.925851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43258 (local address 127.2.32.2:45357)
0621 05:24:54.925918 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:54.925920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:54.925929 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:24:54.925970 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:54.925970 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:54.925993 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:54.926076 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:24:54.926322 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.926856 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.927795 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:54.927988 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:54.928056 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:54.928314 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:24:54.928425 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:24:54.928514 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:54.928598 (+ 84us) server_negotiation.cc:299] Negotiation successful
0621 05:24:54.928606 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 05:24:54.930845 20716 raft_consensus.cc:1261] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35 [term 1 FOLLOWER]: Refusing update from remote peer 293051fc831e494b83aaec34a6a9f8bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:24:54.931031 20448 raft_consensus.cc:1261] T ee65243e7c274a2f9f281581dfadb6a2 P 801385d835164ad1b65f5ebdb111f49b [term 1 FOLLOWER]: Refusing update from remote peer 293051fc831e494b83aaec34a6a9f8bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:24:54.931345 20789 consensus_queue.cc:1035] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6727e1ba04443b582dd7e3c1ebdae35" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:24:54.931525 20783 consensus_queue.cc:1035] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "801385d835164ad1b65f5ebdb111f49b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:24:54.934707 20793 mvcc.cc:205] Tried to move back new op lower bound from 7299151851233959936 to 7299151851108614144. Current Snapshot: MvccSnapshot[applied={T|T < 7299151851233959936}]
I20260621 05:24:54.937628 20303 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:24:54.937747 20303 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:24:54.938736 20303 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08a71040164a44749ca9f9325309354e: Sending DeleteTablet for 3 replicas of tablet ee65243e7c274a2f9f281581dfadb6a2
I20260621 05:24:54.939121 20562 tablet_service.cc:1514] Processing DeleteTablet for tablet ee65243e7c274a2f9f281581dfadb6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:24:54 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:43248
I20260621 05:24:54.939193 20696 tablet_service.cc:1514] Processing DeleteTablet for tablet ee65243e7c274a2f9f281581dfadb6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:24:54 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:55342
I20260621 05:24:54.939276 20428 tablet_service.cc:1514] Processing DeleteTablet for tablet ee65243e7c274a2f9f281581dfadb6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:24:54 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:38274
I20260621 05:24:54.939520 20803 tablet_replica.cc:330] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35: stopping tablet replica
I20260621 05:24:54.939525 20804 tablet_replica.cc:330] T ee65243e7c274a2f9f281581dfadb6a2 P 801385d835164ad1b65f5ebdb111f49b: stopping tablet replica
I20260621 05:24:54.939613 20803 raft_consensus.cc:2229] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:24:54.939678 20804 raft_consensus.cc:2229] T ee65243e7c274a2f9f281581dfadb6a2 P 801385d835164ad1b65f5ebdb111f49b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:24:54.939718 20803 raft_consensus.cc:2258] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:24:54.939797 20804 raft_consensus.cc:2258] T ee65243e7c274a2f9f281581dfadb6a2 P 801385d835164ad1b65f5ebdb111f49b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:24:54.940034 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 20365
I20260621 05:24:54.940220 20803 ts_tablet_manager.cc:1905] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:24:54.942099 20803 ts_tablet_manager.cc:1918] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:24:54.942162 20803 log.cc:1196] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/ee65243e7c274a2f9f281581dfadb6a2
I20260621 05:24:54.942440 20803 ts_tablet_manager.cc:1939] T ee65243e7c274a2f9f281581dfadb6a2 P d6727e1ba04443b582dd7e3c1ebdae35: Deleting consensus metadata
I20260621 05:24:54.942832 20289 catalog_manager.cc:4841] TS d6727e1ba04443b582dd7e3c1ebdae35 (127.2.32.3:34643): tablet ee65243e7c274a2f9f281581dfadb6a2 (table test-table [id=b8eb4fb1a9314f97bcdd0d983c06c70e]) successfully deleted
I20260621 05:24:54.944213 20802 tablet_replica.cc:330] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd: stopping tablet replica
I20260621 05:24:54.944361 20802 raft_consensus.cc:2229] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:24:54.944552 20802 raft_consensus.cc:2258] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:24:54.945021 20802 ts_tablet_manager.cc:1905] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:24:54.947270 20802 ts_tablet_manager.cc:1918] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:24:54.947395 20802 log.cc:1196] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/ee65243e7c274a2f9f281581dfadb6a2
I20260621 05:24:54.947726 20802 ts_tablet_manager.cc:1939] T ee65243e7c274a2f9f281581dfadb6a2 P 293051fc831e494b83aaec34a6a9f8bd: Deleting consensus metadata
W20260621 05:24:54.948045 20287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.32.1:36019 (error 108)
I20260621 05:24:54.948344 20289 catalog_manager.cc:4841] TS 293051fc831e494b83aaec34a6a9f8bd (127.2.32.2:45357): tablet ee65243e7c274a2f9f281581dfadb6a2 (table test-table [id=b8eb4fb1a9314f97bcdd0d983c06c70e]) successfully deleted
I20260621 05:24:54.948638 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:54.948514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:36019 (local address 127.0.0.1:38280)
0621 05:24:54.948559 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:54.948599 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.1:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260621 05:24:54.948740 20287 catalog_manager.cc:4584] TS 801385d835164ad1b65f5ebdb111f49b (127.2.32.1:36019): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee65243e7c274a2f9f281581dfadb6a2: Network error: Client connection negotiation failed: client connection to 127.2.32.1:36019: connect: Connection refused (error 111)
I20260621 05:24:54.948853 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 20499
I20260621 05:24:54.953467 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 20633
I20260621 05:24:54.958951 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 20271
2026-06-21T05:24:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2075 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-21T05:24:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:24:54Z Disabled control of system clock
I20260621 05:24:54.988433 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:34675
--webserver_interface=127.2.32.62
--webserver_port=0
--builtin_ntp_servers=127.2.32.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:34675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:24:55.072258 20809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:55.072556 20809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:55.072592 20809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:55.074678 20809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:24:55.074757 20809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:55.074772 20809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:55.074787 20809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:24:55.074836 20809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:24:55.076784 20809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:34675
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:34675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:55.077021 20809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:55.077301 20809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:55.082170 20814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:55.082207 20817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:55.082371 20815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:55.082686 20809 server_base.cc:1034] running on GCE node
I20260621 05:24:55.082933 20809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:55.083257 20809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:55.084482 20809 hybrid_clock.cc:648] HybridClock initialized: now 1782019495084462 us; error 49 us; skew 500 ppm
I20260621 05:24:55.089838 20809 webserver.cc:460] Webserver started at http://127.2.32.62:43277/ using document root <none> and password file <none>
I20260621 05:24:55.090240 20809 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:55.090353 20809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:55.090515 20809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:55.091706 20809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "dc88a8fdb67b4b1d911d0c8b6906b2da"
format_stamp: "Formatted at 2026-06-21 05:24:55 on dist-test-slave-b357"
server_key: "855fdd5d34acb4b1ab874874cf3a17cc"
server_key_iv: "0268ca312e7888fb59321de62de17675"
server_key_version: "encryptionkey@0"
I20260621 05:24:55.092097 20809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "dc88a8fdb67b4b1d911d0c8b6906b2da"
format_stamp: "Formatted at 2026-06-21 05:24:55 on dist-test-slave-b357"
server_key: "855fdd5d34acb4b1ab874874cf3a17cc"
server_key_iv: "0268ca312e7888fb59321de62de17675"
server_key_version: "encryptionkey@0"
I20260621 05:24:55.093407 20809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:24:55.094228 20823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:55.094422 20809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:55.094530 20809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "dc88a8fdb67b4b1d911d0c8b6906b2da"
format_stamp: "Formatted at 2026-06-21 05:24:55 on dist-test-slave-b357"
server_key: "855fdd5d34acb4b1ab874874cf3a17cc"
server_key_iv: "0268ca312e7888fb59321de62de17675"
server_key_version: "encryptionkey@0"
I20260621 05:24:55.094604 20809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:55.108656 20809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:55.108978 20809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:55.109124 20809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:55.113375 20809 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:34675
I20260621 05:24:55.113411 20875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:34675 every 8 connection(s)
I20260621 05:24:55.113744 20809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:24:55.114022 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 20809
I20260621 05:24:55.114318 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:24:55.114565 2176 external_mini_cluster.cc:1428] Setting key af75f7771e869e9b81ad625ee5103de6
I20260621 05:24:55.114854 20876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:55.118317 20876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da: Bootstrap starting.
I20260621 05:24:55.118952 20876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:55.119280 20876 log.cc:824] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:55.120385 20876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da: No bootstrap required, opened a new log
I20260621 05:24:55.121273 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:55.115116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44466 (local address 127.2.32.62:34675)
0621 05:24:55.115366 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:55.115374 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:55.115405 (+ 31us) server_negotiation.cc:407] Connection header received
0621 05:24:55.116017 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:55.116026 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:55.116103 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:55.116335 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:55.116490 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.117816 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.118734 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.119376 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.119499 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:55.120597 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:55.120602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:55.120602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:24:55.120796 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:55.120836 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:24:55.120849 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:55.120920 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:55.120985 (+ 65us) server_negotiation.cc:299] Negotiation successful
0621 05:24:55.121064 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
I20260621 05:24:55.122581 20876 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc88a8fdb67b4b1d911d0c8b6906b2da" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34675 } }
I20260621 05:24:55.122712 20876 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:55.122735 20876 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc88a8fdb67b4b1d911d0c8b6906b2da, State: Initialized, Role: FOLLOWER
I20260621 05:24:55.122864 20876 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc88a8fdb67b4b1d911d0c8b6906b2da" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34675 } }
I20260621 05:24:55.122937 20876 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:24:55.122959 20876 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:24:55.123008 20876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:55.123637 20876 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc88a8fdb67b4b1d911d0c8b6906b2da" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34675 } }
I20260621 05:24:55.123721 20876 leader_election.cc:302] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc88a8fdb67b4b1d911d0c8b6906b2da; no voters:
I20260621 05:24:55.123878 20876 leader_election.cc:288] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:24:55.123963 20881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:24:55.124123 20876 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:24:55.124150 20881 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da [term 1 LEADER]: Becoming Leader. State: Replica: dc88a8fdb67b4b1d911d0c8b6906b2da, State: Running, Role: LEADER
I20260621 05:24:55.124224 20881 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc88a8fdb67b4b1d911d0c8b6906b2da" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34675 } }
I20260621 05:24:55.124608 20883 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc88a8fdb67b4b1d911d0c8b6906b2da. Latest consensus state: current_term: 1 leader_uuid: "dc88a8fdb67b4b1d911d0c8b6906b2da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc88a8fdb67b4b1d911d0c8b6906b2da" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34675 } } }
I20260621 05:24:55.124713 20883 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da [sys.catalog]: This master's current role is: LEADER
I20260621 05:24:55.124917 20890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:24:55.124589 20882 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc88a8fdb67b4b1d911d0c8b6906b2da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc88a8fdb67b4b1d911d0c8b6906b2da" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34675 } } }
I20260621 05:24:55.125100 20882 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da [sys.catalog]: This master's current role is: LEADER
I20260621 05:24:55.125381 20890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:24:55.126690 20890 catalog_manager.cc:1347] Generated new cluster ID: 6379f10ae60d405187a0837ff3ae666d
I20260621 05:24:55.126741 20890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:24:55.139695 20890 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:24:55.140174 20890 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:24:55.146873 20890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da: Generated new TSK 0
I20260621 05:24:55.147171 20890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:24:55.150110 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:34675
--builtin_ntp_servers=127.2.32.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:24:55.232332 20900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:55.232517 20900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:55.232537 20900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:55.233626 20900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:55.233665 20900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:55.233690 20900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:24:55.234968 20900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:34675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:55.235194 20900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:55.235500 20900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 05:24:55.240787 20900 server_base.cc:1034] running on GCE node
W20260621 05:24:55.240633 20905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:55.240649 20906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:55.240631 20908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:55.241333 20900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:55.241621 20900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:55.242803 20900 hybrid_clock.cc:648] HybridClock initialized: now 1782019495242780 us; error 45 us; skew 500 ppm
I20260621 05:24:55.244369 20900 webserver.cc:460] Webserver started at http://127.2.32.1:32963/ using document root <none> and password file <none>
I20260621 05:24:55.244700 20900 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:55.244779 20900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:55.244939 20900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:55.246219 20900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "36f71891d12f4043903a73a5d35d1eeb"
format_stamp: "Formatted at 2026-06-21 05:24:55 on dist-test-slave-b357"
server_key: "769c19a67287f11f1ff1890b3adcaeb0"
server_key_iv: "89e6de7052b93fbf95c74c5428f06fc3"
server_key_version: "encryptionkey@0"
I20260621 05:24:55.246629 20900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "36f71891d12f4043903a73a5d35d1eeb"
format_stamp: "Formatted at 2026-06-21 05:24:55 on dist-test-slave-b357"
server_key: "769c19a67287f11f1ff1890b3adcaeb0"
server_key_iv: "89e6de7052b93fbf95c74c5428f06fc3"
server_key_version: "encryptionkey@0"
I20260621 05:24:55.248217 20900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:24:55.249282 20914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:55.249534 20900 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:55.249646 20900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "36f71891d12f4043903a73a5d35d1eeb"
format_stamp: "Formatted at 2026-06-21 05:24:55 on dist-test-slave-b357"
server_key: "769c19a67287f11f1ff1890b3adcaeb0"
server_key_iv: "89e6de7052b93fbf95c74c5428f06fc3"
server_key_version: "encryptionkey@0"
I20260621 05:24:55.249727 20900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:55.281234 20900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:55.281503 20900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:55.281603 20900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:55.281796 20900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:55.282063 20900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:55.282094 20900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:55.282115 20900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:55.282128 20900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:55.287968 20900 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:36905
I20260621 05:24:55.288035 21027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:36905 every 8 connection(s)
I20260621 05:24:55.288357 20900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:24:55.294462 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:55.289061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:36303 (local address 127.2.32.62:34675)
0621 05:24:55.289137 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:55.289140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:55.289664 (+ 524us) server_negotiation.cc:407] Connection header received
0621 05:24:55.290298 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:55.290300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:55.290319 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:55.290409 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:55.291221 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.291724 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.292980 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.293219 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.293303 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:55.294132 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:55.294134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:55.294134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:24:55.294139 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:55.294155 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:24:55.294159 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:55.294221 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:55.294370 (+ 149us) server_negotiation.cc:299] Negotiation successful
0621 05:24:55.294404 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 05:24:55.294652 21030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:55.289038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:34675 (local address 127.2.32.1:36303)
0621 05:24:55.289350 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:55.289416 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:55.290141 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:55.290452 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:55.290456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:55.290465 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:55.291049 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:55.291053 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.291842 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.291843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:55.292832 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:55.292838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.293333 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.293335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:55.293460 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:55.293946 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:24:55.293947 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:55.293988 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:24:55.294018 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:55.294251 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:55.294257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:55.294259 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:24:55.294338 (+ 79us) client_negotiation.cc:240] Negotiation successful
0621 05:24:55.294434 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
I20260621 05:24:55.295197 21028 heartbeater.cc:347] Connected to a master server at 127.2.32.62:34675
I20260621 05:24:55.295298 21028 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:55.295471 21028 heartbeater.cc:511] Master 127.2.32.62:34675 requested a full tablet report, sending...
I20260621 05:24:55.296239 20840 ts_manager.cc:194] Registered new tserver with Master: 36f71891d12f4043903a73a5d35d1eeb (127.2.32.1:36905)
I20260621 05:24:55.296409 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 20900
I20260621 05:24:55.296531 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:24:55.296734 2176 external_mini_cluster.cc:1428] Setting key 5cb6338c58addb3535dba32110f6849a
I20260621 05:24:55.297098 20840 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.32.1:36303
I20260621 05:24:55.297847 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:34675
--builtin_ntp_servers=127.2.32.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:24:55.383711 21031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:55.383966 21031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:55.383992 21031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:55.386096 21031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:55.386161 21031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:55.386196 21031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:24:55.388625 21031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:34675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:55.388911 21031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:55.389237 21031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:55.395197 21039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:55.395254 21036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:55.395197 21037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:55.395475 21031 server_base.cc:1034] running on GCE node
I20260621 05:24:55.395738 21031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:55.396014 21031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:55.397200 21031 hybrid_clock.cc:648] HybridClock initialized: now 1782019495397191 us; error 43 us; skew 500 ppm
I20260621 05:24:55.398625 21031 webserver.cc:460] Webserver started at http://127.2.32.2:46247/ using document root <none> and password file <none>
I20260621 05:24:55.398880 21031 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:55.398947 21031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:55.399092 21031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:55.400192 21031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "5ddf2fd7397a4ffaae881968bc7da2e0"
format_stamp: "Formatted at 2026-06-21 05:24:55 on dist-test-slave-b357"
server_key: "5abdb6f0714001d84957104ecf8175e3"
server_key_iv: "812de65a45de125232ca32a6aa05a939"
server_key_version: "encryptionkey@0"
I20260621 05:24:55.400665 21031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "5ddf2fd7397a4ffaae881968bc7da2e0"
format_stamp: "Formatted at 2026-06-21 05:24:55 on dist-test-slave-b357"
server_key: "5abdb6f0714001d84957104ecf8175e3"
server_key_iv: "812de65a45de125232ca32a6aa05a939"
server_key_version: "encryptionkey@0"
I20260621 05:24:55.402060 21031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:24:55.402992 21045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:55.403225 21031 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:55.403331 21031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "5ddf2fd7397a4ffaae881968bc7da2e0"
format_stamp: "Formatted at 2026-06-21 05:24:55 on dist-test-slave-b357"
server_key: "5abdb6f0714001d84957104ecf8175e3"
server_key_iv: "812de65a45de125232ca32a6aa05a939"
server_key_version: "encryptionkey@0"
I20260621 05:24:55.403411 21031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:55.431116 21031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:55.431559 21031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:55.431738 21031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:55.432026 21031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:55.432427 21031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:55.432483 21031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:55.432529 21031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:55.432569 21031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:55.440237 21031 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:46461
I20260621 05:24:55.440281 21158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:46461 every 8 connection(s)
I20260621 05:24:55.440657 21031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:24:55.443358 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 21031
I20260621 05:24:55.443509 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:24:55.443802 2176 external_mini_cluster.cc:1428] Setting key 70979cda5b6a2bf2637d3a64e5ab5fc9
I20260621 05:24:55.445225 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:34675
--builtin_ntp_servers=127.2.32.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 05:24:55.446763 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:55.441514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:40179 (local address 127.2.32.62:34675)
0621 05:24:55.441623 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:55.441626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:55.441797 (+ 171us) server_negotiation.cc:407] Connection header received
0621 05:24:55.442318 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:55.442319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:55.442330 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:55.442406 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:55.443118 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.443730 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.444866 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.445244 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.445344 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:55.446390 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:55.446391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:55.446392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:24:55.446397 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:55.446419 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:24:55.446424 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:55.446491 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:55.446592 (+ 101us) server_negotiation.cc:299] Negotiation successful
0621 05:24:55.446627 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 05:24:55.446800 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:55.441431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:34675 (local address 127.2.32.2:40179)
0621 05:24:55.441667 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:55.441709 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:55.442215 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:55.442477 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:55.442480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:55.442488 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:55.443012 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:55.443016 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.443814 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.443815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:55.444748 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:55.444752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.445382 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.445384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:55.445556 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:55.446241 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:24:55.446241 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:55.446268 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:24:55.446295 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:55.446499 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:55.446502 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:55.446504 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:24:55.446573 (+ 69us) client_negotiation.cc:240] Negotiation successful
0621 05:24:55.446646 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":90,"threads_started":1}
I20260621 05:24:55.447342 21159 heartbeater.cc:347] Connected to a master server at 127.2.32.62:34675
I20260621 05:24:55.447451 21159 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:55.447626 21159 heartbeater.cc:511] Master 127.2.32.62:34675 requested a full tablet report, sending...
I20260621 05:24:55.448041 20839 ts_manager.cc:194] Registered new tserver with Master: 5ddf2fd7397a4ffaae881968bc7da2e0 (127.2.32.2:46461)
I20260621 05:24:55.448421 20839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.32.2:40179
W20260621 05:24:55.534224 21162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:55.534453 21162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:55.534484 21162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:55.535656 21162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:55.535709 21162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:55.535753 21162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:24:55.537171 21162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:34675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:55.537426 21162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:55.537705 21162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:55.542656 21167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:55.542685 21168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:55.542698 21170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:55.542949 21162 server_base.cc:1034] running on GCE node
I20260621 05:24:55.543183 21162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:55.543416 21162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:55.544598 21162 hybrid_clock.cc:648] HybridClock initialized: now 1782019495544549 us; error 64 us; skew 500 ppm
I20260621 05:24:55.545881 21162 webserver.cc:460] Webserver started at http://127.2.32.3:39231/ using document root <none> and password file <none>
I20260621 05:24:55.546139 21162 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:55.546205 21162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:55.546342 21162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:55.547474 21162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "4da8443b3bd548579f716feb815c7966"
format_stamp: "Formatted at 2026-06-21 05:24:55 on dist-test-slave-b357"
server_key: "41a65233c7d2f8d8ff7474899655f4e1"
server_key_iv: "a3b936d27064c4f3f62d635f78be3048"
server_key_version: "encryptionkey@0"
I20260621 05:24:55.547837 21162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "4da8443b3bd548579f716feb815c7966"
format_stamp: "Formatted at 2026-06-21 05:24:55 on dist-test-slave-b357"
server_key: "41a65233c7d2f8d8ff7474899655f4e1"
server_key_iv: "a3b936d27064c4f3f62d635f78be3048"
server_key_version: "encryptionkey@0"
I20260621 05:24:55.549129 21162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:24:55.549916 21176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:55.550272 21162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:55.550400 21162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "4da8443b3bd548579f716feb815c7966"
format_stamp: "Formatted at 2026-06-21 05:24:55 on dist-test-slave-b357"
server_key: "41a65233c7d2f8d8ff7474899655f4e1"
server_key_iv: "a3b936d27064c4f3f62d635f78be3048"
server_key_version: "encryptionkey@0"
I20260621 05:24:55.550500 21162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:55.564576 21162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:55.564898 21162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:55.565047 21162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:55.565269 21162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:55.565584 21162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:55.565635 21162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:55.565678 21162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:55.565703 21162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:55.571712 21162 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:34081
I20260621 05:24:55.571789 21289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:34081 every 8 connection(s)
I20260621 05:24:55.572096 21162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:24:55.577509 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:55.572934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:46231 (local address 127.2.32.62:34675)
0621 05:24:55.572982 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:55.572984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:55.573364 (+ 380us) server_negotiation.cc:407] Connection header received
0621 05:24:55.573907 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:55.573908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:55.573920 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:55.573967 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:55.574638 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.575163 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.576122 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.576314 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.576388 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:55.577217 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:55.577218 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:55.577219 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:24:55.577223 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:55.577238 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:24:55.577242 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:55.577308 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:55.577396 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 05:24:55.577426 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:24:55.577790 21292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:55.572926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:34675 (local address 127.2.32.3:46231)
0621 05:24:55.573211 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:55.573248 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:55.573805 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:55.574032 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:55.574036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:55.574045 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:55.574550 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:55.574554 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.575258 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.575259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:55.576024 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:55.576026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.576395 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.576396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:55.576528 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:55.577069 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:24:55.577069 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:55.577095 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:24:55.577117 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:55.577311 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:55.577316 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:55.577319 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:24:55.577397 (+ 78us) client_negotiation.cc:240] Negotiation successful
0621 05:24:55.577493 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":154,"threads_started":1}
I20260621 05:24:55.578293 21290 heartbeater.cc:347] Connected to a master server at 127.2.32.62:34675
I20260621 05:24:55.578370 21290 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:55.578552 21290 heartbeater.cc:511] Master 127.2.32.62:34675 requested a full tablet report, sending...
I20260621 05:24:55.578933 20839 ts_manager.cc:194] Registered new tserver with Master: 4da8443b3bd548579f716feb815c7966 (127.2.32.3:34081)
I20260621 05:24:55.579348 20839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.32.3:46231
I20260621 05:24:55.582340 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 21162
I20260621 05:24:55.582475 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:24:55.582754 2176 external_mini_cluster.cc:1428] Setting key 6b8c7819edf8d2f2d55e5ea3bc7fdecb
I20260621 05:24:55.583724 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:24:55.588621 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:55.585058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44476 (local address 127.2.32.62:34675)
0621 05:24:55.585099 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:55.585101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:55.585219 (+ 118us) server_negotiation.cc:407] Connection header received
0621 05:24:55.585326 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:55.585328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:55.585342 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:55.585435 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:55.585745 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.586207 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.587002 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.587251 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.587302 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:55.588164 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:55.588166 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:55.588166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:24:55.588171 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:55.588192 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:24:55.588196 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:55.588301 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:55.588429 (+ 128us) server_negotiation.cc:299] Negotiation successful
0621 05:24:55.588448 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:24:55.590592 20839 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:24:55.592454 20839 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:24:55.599805 21302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:55.595477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49686 (local address 127.2.32.3:34081)
0621 05:24:55.595947 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:55.595952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:55.595964 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:24:55.596015 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:55.596017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:55.596037 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:55.596142 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:55.596384 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.597424 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.598547 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.599389 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.599464 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:55.599553 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:55.599670 (+ 117us) server_negotiation.cc:299] Negotiation successful
0621 05:24:55.599718 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":142,"threads_started":1}
I20260621 05:24:55.599838 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:55.595298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:34081 (local address 127.0.0.1:49686)
0621 05:24:55.595657 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:55.595670 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:55.595740 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:55.596158 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:55.596159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:55.596161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:55.596230 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:55.596232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.597507 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.597508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:55.598465 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:55.598467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.599478 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.599479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:55.599554 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:55.599567 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 05:24:55.599647 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 05:24:55.599704 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":1,"thread_start_us":87,"threads_started":1}
I20260621 05:24:55.600082 21298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:55.595504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:46461 (local address 127.0.0.1:53200)
0621 05:24:55.596245 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:55.596311 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:55.596391 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:55.596638 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:55.596638 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:55.596641 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:55.596692 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:55.596694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.597998 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.597999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:55.598991 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:55.598993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.599879 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.599880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:55.599939 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:55.599943 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:24:55.599993 (+ 50us) client_negotiation.cc:240] Negotiation successful
0621 05:24:55.600027 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":102,"threads_started":1}
I20260621 05:24:55.600239 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:55.595329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54658 (local address 127.2.32.1:36905)
0621 05:24:55.595545 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:55.595552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:55.595735 (+ 183us) server_negotiation.cc:407] Connection header received
0621 05:24:55.595789 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:55.595792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:55.595811 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:55.595885 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:55.596159 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.597221 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.598683 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.599389 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.599462 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:55.599552 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:55.600043 (+ 491us) server_negotiation.cc:299] Negotiation successful
0621 05:24:55.600083 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":89,"threads_started":1}
I20260621 05:24:55.600349 21303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:55.595755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.2.32.2:46461)
0621 05:24:55.596040 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:55.596044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:55.596467 (+ 423us) server_negotiation.cc:407] Connection header received
0621 05:24:55.596514 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:55.596516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:55.596531 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:55.596632 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:55.596752 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.597869 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.599066 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.599704 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.599764 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:55.599845 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:55.600221 (+ 376us) server_negotiation.cc:299] Negotiation successful
0621 05:24:55.600263 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":96,"threads_started":1}
I20260621 05:24:55.600721 21224 tablet_service.cc:1467] Processing CreateTablet for tablet 8de975dfc13642d7a8fdf11fa9473c8d (DEFAULT_TABLE table=test-table [id=299cadcf73bc47d7a86fd01162c4d0b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:55.600900 21093 tablet_service.cc:1467] Processing CreateTablet for tablet 8de975dfc13642d7a8fdf11fa9473c8d (DEFAULT_TABLE table=test-table [id=299cadcf73bc47d7a86fd01162c4d0b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:55.601047 21224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8de975dfc13642d7a8fdf11fa9473c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:55.601208 21093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8de975dfc13642d7a8fdf11fa9473c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:55.603834 21304 tablet_bootstrap.cc:492] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966: Bootstrap starting.
I20260621 05:24:55.603868 21299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:55.595236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:36905 (local address 127.0.0.1:54658)
0621 05:24:55.595518 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:55.595563 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:55.595643 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:55.595999 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:55.596002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:55.596005 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:55.596073 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:55.596075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.597338 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.597339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:55.598599 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:55.598601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.599589 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.599590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:55.599670 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:55.599675 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:24:55.599726 (+ 51us) client_negotiation.cc:240] Negotiation successful
0621 05:24:55.599764 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":3,"thread_start_us":117,"threads_started":1}
I20260621 05:24:55.604533 20962 tablet_service.cc:1467] Processing CreateTablet for tablet 8de975dfc13642d7a8fdf11fa9473c8d (DEFAULT_TABLE table=test-table [id=299cadcf73bc47d7a86fd01162c4d0b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:55.604655 21304 tablet_bootstrap.cc:654] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:55.604866 20962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8de975dfc13642d7a8fdf11fa9473c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:55.604991 21304 log.cc:824] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:55.605321 21305 tablet_bootstrap.cc:492] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0: Bootstrap starting.
I20260621 05:24:55.605934 21305 tablet_bootstrap.cc:654] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:55.606078 21304 tablet_bootstrap.cc:492] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966: No bootstrap required, opened a new log
I20260621 05:24:55.606187 21304 ts_tablet_manager.cc:1397] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:24:55.606253 21305 log.cc:824] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:55.607996 21308 tablet_bootstrap.cc:492] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb: Bootstrap starting.
I20260621 05:24:55.608500 21305 tablet_bootstrap.cc:492] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0: No bootstrap required, opened a new log
I20260621 05:24:55.608556 21305 ts_tablet_manager.cc:1397] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260621 05:24:55.608913 21304 raft_consensus.cc:348] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f71891d12f4043903a73a5d35d1eeb" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36905 } } peers { permanent_uuid: "4da8443b3bd548579f716feb815c7966" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34081 } } peers { permanent_uuid: "5ddf2fd7397a4ffaae881968bc7da2e0" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46461 } }
I20260621 05:24:55.609102 21304 raft_consensus.cc:374] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:55.609145 21304 raft_consensus.cc:729] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da8443b3bd548579f716feb815c7966, State: Initialized, Role: FOLLOWER
I20260621 05:24:55.609283 21304 consensus_queue.cc:260] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f71891d12f4043903a73a5d35d1eeb" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36905 } } peers { permanent_uuid: "4da8443b3bd548579f716feb815c7966" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34081 } } peers { permanent_uuid: "5ddf2fd7397a4ffaae881968bc7da2e0" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46461 } }
I20260621 05:24:55.609584 21304 ts_tablet_manager.cc:1428] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 05:24:55.609680 21290 heartbeater.cc:503] Master 127.2.32.62:34675 was elected leader, sending a full tablet report...
I20260621 05:24:55.610692 21305 raft_consensus.cc:348] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f71891d12f4043903a73a5d35d1eeb" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36905 } } peers { permanent_uuid: "4da8443b3bd548579f716feb815c7966" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34081 } } peers { permanent_uuid: "5ddf2fd7397a4ffaae881968bc7da2e0" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46461 } }
I20260621 05:24:55.610893 21305 raft_consensus.cc:374] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:55.610948 21305 raft_consensus.cc:729] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ddf2fd7397a4ffaae881968bc7da2e0, State: Initialized, Role: FOLLOWER
I20260621 05:24:55.611181 21305 consensus_queue.cc:260] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f71891d12f4043903a73a5d35d1eeb" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36905 } } peers { permanent_uuid: "4da8443b3bd548579f716feb815c7966" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34081 } } peers { permanent_uuid: "5ddf2fd7397a4ffaae881968bc7da2e0" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46461 } }
I20260621 05:24:55.611464 21305 ts_tablet_manager.cc:1428] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 05:24:55.611555 21159 heartbeater.cc:503] Master 127.2.32.62:34675 was elected leader, sending a full tablet report...
I20260621 05:24:55.611994 21308 tablet_bootstrap.cc:654] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:55.612310 21308 log.cc:824] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:55.613368 21308 tablet_bootstrap.cc:492] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb: No bootstrap required, opened a new log
I20260621 05:24:55.613427 21308 ts_tablet_manager.cc:1397] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260621 05:24:55.615170 21308 raft_consensus.cc:348] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f71891d12f4043903a73a5d35d1eeb" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36905 } } peers { permanent_uuid: "4da8443b3bd548579f716feb815c7966" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34081 } } peers { permanent_uuid: "5ddf2fd7397a4ffaae881968bc7da2e0" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46461 } }
I20260621 05:24:55.615331 21308 raft_consensus.cc:374] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:55.615375 21308 raft_consensus.cc:729] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f71891d12f4043903a73a5d35d1eeb, State: Initialized, Role: FOLLOWER
I20260621 05:24:55.615480 21308 consensus_queue.cc:260] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f71891d12f4043903a73a5d35d1eeb" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36905 } } peers { permanent_uuid: "4da8443b3bd548579f716feb815c7966" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34081 } } peers { permanent_uuid: "5ddf2fd7397a4ffaae881968bc7da2e0" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46461 } }
I20260621 05:24:55.615713 21308 ts_tablet_manager.cc:1428] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 05:24:55.615782 21028 heartbeater.cc:503] Master 127.2.32.62:34675 was elected leader, sending a full tablet report...
I20260621 05:24:55.616653 21312 raft_consensus.cc:482] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:24:55.616736 21312 raft_consensus.cc:504] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f71891d12f4043903a73a5d35d1eeb" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36905 } } peers { permanent_uuid: "4da8443b3bd548579f716feb815c7966" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34081 } } peers { permanent_uuid: "5ddf2fd7397a4ffaae881968bc7da2e0" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46461 } }
I20260621 05:24:55.617012 21312 leader_election.cc:288] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da8443b3bd548579f716feb815c7966 (127.2.32.3:34081), 5ddf2fd7397a4ffaae881968bc7da2e0 (127.2.32.2:46461)
I20260621 05:24:55.621098 21303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:55.617394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:42691 (local address 127.2.32.2:46461)
0621 05:24:55.617432 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:55.617434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:55.617620 (+ 186us) server_negotiation.cc:407] Connection header received
0621 05:24:55.617673 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:55.617673 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:55.617700 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:55.617764 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:55.617889 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.618572 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.620116 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.620709 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.620802 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:55.620811 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:55.621001 (+ 190us) server_negotiation.cc:299] Negotiation successful
0621 05:24:55.621037 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:24:55.621098 21302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:55.617250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:40765 (local address 127.2.32.3:34081)
0621 05:24:55.617437 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:55.617440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:55.617450 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:24:55.617479 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:55.617480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:55.617491 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:55.617551 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:55.617722 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.618518 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.620116 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.620711 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.620798 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:55.620808 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:55.620993 (+ 185us) server_negotiation.cc:299] Negotiation successful
0621 05:24:55.621037 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260621 05:24:55.621132 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:55.617320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:46461 (local address 127.2.32.1:42691)
0621 05:24:55.617511 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:55.617538 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:55.617607 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:55.617769 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:55.617769 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:55.617771 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:55.617821 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:55.617822 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.618655 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.618656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:55.620040 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:55.620042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.620816 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.620817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:55.620921 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:55.620924 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:24:55.620979 (+ 55us) client_negotiation.cc:240] Negotiation successful
0621 05:24:55.621035 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":49,"thread_start_us":130,"threads_started":1}
I20260621 05:24:55.621140 21030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:55.617177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:34081 (local address 127.2.32.1:40765)
0621 05:24:55.617270 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:55.617284 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:55.617335 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:55.617576 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:55.617577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:55.617579 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:55.617644 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:55.617646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.618651 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.618652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:55.620001 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:55.620003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.620814 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.620815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:55.620903 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:55.620907 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:24:55.620979 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 05:24:55.621038 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":82}
I20260621 05:24:55.621754 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de975dfc13642d7a8fdf11fa9473c8d" candidate_uuid: "36f71891d12f4043903a73a5d35d1eeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da8443b3bd548579f716feb815c7966" is_pre_election: true
I20260621 05:24:55.621770 21113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de975dfc13642d7a8fdf11fa9473c8d" candidate_uuid: "36f71891d12f4043903a73a5d35d1eeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ddf2fd7397a4ffaae881968bc7da2e0" is_pre_election: true
I20260621 05:24:55.621887 21244 raft_consensus.cc:2454] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36f71891d12f4043903a73a5d35d1eeb in term 0.
I20260621 05:24:55.621893 21113 raft_consensus.cc:2454] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36f71891d12f4043903a73a5d35d1eeb in term 0.
I20260621 05:24:55.622089 20916 leader_election.cc:302] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36f71891d12f4043903a73a5d35d1eeb, 5ddf2fd7397a4ffaae881968bc7da2e0; no voters:
I20260621 05:24:55.622239 21312 raft_consensus.cc:2781] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:24:55.622300 21312 raft_consensus.cc:482] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:24:55.622318 21312 raft_consensus.cc:3037] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:55.622947 21312 raft_consensus.cc:504] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f71891d12f4043903a73a5d35d1eeb" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36905 } } peers { permanent_uuid: "4da8443b3bd548579f716feb815c7966" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34081 } } peers { permanent_uuid: "5ddf2fd7397a4ffaae881968bc7da2e0" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46461 } }
I20260621 05:24:55.623086 21312 leader_election.cc:288] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [CANDIDATE]: Term 1 election: Requested vote from peers 4da8443b3bd548579f716feb815c7966 (127.2.32.3:34081), 5ddf2fd7397a4ffaae881968bc7da2e0 (127.2.32.2:46461)
I20260621 05:24:55.623235 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de975dfc13642d7a8fdf11fa9473c8d" candidate_uuid: "36f71891d12f4043903a73a5d35d1eeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da8443b3bd548579f716feb815c7966"
I20260621 05:24:55.623322 21244 raft_consensus.cc:3037] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:55.623333 21113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de975dfc13642d7a8fdf11fa9473c8d" candidate_uuid: "36f71891d12f4043903a73a5d35d1eeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ddf2fd7397a4ffaae881968bc7da2e0"
I20260621 05:24:55.623431 21113 raft_consensus.cc:3037] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:55.624073 21244 raft_consensus.cc:2454] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36f71891d12f4043903a73a5d35d1eeb in term 1.
I20260621 05:24:55.624225 20916 leader_election.cc:302] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36f71891d12f4043903a73a5d35d1eeb, 4da8443b3bd548579f716feb815c7966; no voters:
I20260621 05:24:55.624244 21113 raft_consensus.cc:2454] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36f71891d12f4043903a73a5d35d1eeb in term 1.
I20260621 05:24:55.624328 21312 raft_consensus.cc:2781] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:24:55.624498 21312 raft_consensus.cc:686] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [term 1 LEADER]: Becoming Leader. State: Replica: 36f71891d12f4043903a73a5d35d1eeb, State: Running, Role: LEADER
I20260621 05:24:55.624590 21312 consensus_queue.cc:237] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f71891d12f4043903a73a5d35d1eeb" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36905 } } peers { permanent_uuid: "4da8443b3bd548579f716feb815c7966" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34081 } } peers { permanent_uuid: "5ddf2fd7397a4ffaae881968bc7da2e0" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46461 } }
I20260621 05:24:55.625375 20840 catalog_manager.cc:5495] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 36f71891d12f4043903a73a5d35d1eeb (127.2.32.1). New cstate: current_term: 1 leader_uuid: "36f71891d12f4043903a73a5d35d1eeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f71891d12f4043903a73a5d35d1eeb" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da8443b3bd548579f716feb815c7966" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ddf2fd7397a4ffaae881968bc7da2e0" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46461 } health_report { overall_health: UNKNOWN } } }
I20260621 05:24:55.635603 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:55.632915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54662 (local address 127.2.32.1:36905)
0621 05:24:55.632952 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:55.632954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:55.632961 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:24:55.632991 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:55.632991 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:55.633013 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:55.633075 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:24:55.633249 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.633693 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.634644 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:55.634899 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:55.634971 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:55.635276 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:24:55.635365 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:24:55.635449 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:55.635533 (+ 84us) server_negotiation.cc:299] Negotiation successful
0621 05:24:55.635541 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:24:55.637769 21113 raft_consensus.cc:1261] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0 [term 1 FOLLOWER]: Refusing update from remote peer 36f71891d12f4043903a73a5d35d1eeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:24:55.637796 21244 raft_consensus.cc:1261] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966 [term 1 FOLLOWER]: Refusing update from remote peer 36f71891d12f4043903a73a5d35d1eeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:24:55.638130 21314 consensus_queue.cc:1035] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ddf2fd7397a4ffaae881968bc7da2e0" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:24:55.638250 21312 consensus_queue.cc:1035] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da8443b3bd548579f716feb815c7966" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 34081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:24:55.641439 21319 mvcc.cc:205] Tried to move back new op lower bound from 7299151854129840128 to 7299151854079074304. Current Snapshot: MvccSnapshot[applied={T|T < 7299151854129840128 or (T in {7299151854129840128})}]
I20260621 05:24:55.642901 20839 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:24:55.642983 20839 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:24:55.643704 20839 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc88a8fdb67b4b1d911d0c8b6906b2da: Sending DeleteTablet for 3 replicas of tablet 8de975dfc13642d7a8fdf11fa9473c8d
I20260621 05:24:55.644027 20962 tablet_service.cc:1514] Processing DeleteTablet for tablet 8de975dfc13642d7a8fdf11fa9473c8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:24:55 UTC) from {username='slave'} at 127.0.0.1:54658
I20260621 05:24:55.644243 21327 tablet_replica.cc:330] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb: stopping tablet replica
I20260621 05:24:55.644248 21224 tablet_service.cc:1514] Processing DeleteTablet for tablet 8de975dfc13642d7a8fdf11fa9473c8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:24:55 UTC) from {username='slave'} at 127.0.0.1:49686
I20260621 05:24:55.644349 21327 raft_consensus.cc:2229] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:24:55.644382 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 20900
I20260621 05:24:55.644445 21327 raft_consensus.cc:2258] T 8de975dfc13642d7a8fdf11fa9473c8d P 36f71891d12f4043903a73a5d35d1eeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:24:55.644445 21093 tablet_service.cc:1514] Processing DeleteTablet for tablet 8de975dfc13642d7a8fdf11fa9473c8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:24:55 UTC) from {username='slave'} at 127.0.0.1:53200
I20260621 05:24:55.644630 21328 tablet_replica.cc:330] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966: stopping tablet replica
I20260621 05:24:55.644703 21328 raft_consensus.cc:2229] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:24:55.644788 21328 raft_consensus.cc:2258] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:24:55.644917 21329 tablet_replica.cc:330] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0: stopping tablet replica
I20260621 05:24:55.644991 21329 raft_consensus.cc:2229] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:24:55.645071 21329 raft_consensus.cc:2258] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:24:55.645496 21329 ts_tablet_manager.cc:1905] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:24:55.645496 21328 ts_tablet_manager.cc:1905] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:24:55.647161 21328 ts_tablet_manager.cc:1918] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:24:55.647161 21329 ts_tablet_manager.cc:1918] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:24:55.647225 21329 log.cc:1196] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/8de975dfc13642d7a8fdf11fa9473c8d
I20260621 05:24:55.647225 21328 log.cc:1196] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/8de975dfc13642d7a8fdf11fa9473c8d
I20260621 05:24:55.647485 21328 ts_tablet_manager.cc:1939] T 8de975dfc13642d7a8fdf11fa9473c8d P 4da8443b3bd548579f716feb815c7966: Deleting consensus metadata
I20260621 05:24:55.647485 21329 ts_tablet_manager.cc:1939] T 8de975dfc13642d7a8fdf11fa9473c8d P 5ddf2fd7397a4ffaae881968bc7da2e0: Deleting consensus metadata
I20260621 05:24:55.647848 20825 catalog_manager.cc:4841] TS 5ddf2fd7397a4ffaae881968bc7da2e0 (127.2.32.2:46461): tablet 8de975dfc13642d7a8fdf11fa9473c8d (table test-table [id=299cadcf73bc47d7a86fd01162c4d0b6]) successfully deleted
I20260621 05:24:55.647936 20825 catalog_manager.cc:4841] TS 4da8443b3bd548579f716feb815c7966 (127.2.32.3:34081): tablet 8de975dfc13642d7a8fdf11fa9473c8d (table test-table [id=299cadcf73bc47d7a86fd01162c4d0b6]) successfully deleted
W20260621 05:24:55.651064 20826 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.32.1:36905 (error 108)
I20260621 05:24:55.651216 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 21031
I20260621 05:24:55.652588 21299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:55.651655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:36905 (local address 127.0.0.1:54672)
0621 05:24:55.652499 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:55.652548 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.1:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":820}
W20260621 05:24:55.652771 20826 catalog_manager.cc:4584] TS 36f71891d12f4043903a73a5d35d1eeb (127.2.32.1:36905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8de975dfc13642d7a8fdf11fa9473c8d: Network error: Client connection negotiation failed: client connection to 127.2.32.1:36905: connect: Connection refused (error 111)
I20260621 05:24:55.656375 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 21162
I20260621 05:24:55.660935 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 20809
2026-06-21T05:24:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (700 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:24:55 dist-test-slave-b357 krb5kdc[21333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:24:55 dist-test-slave-b357 krb5kdc[21333](info): set up 2 sockets
Jun 21 05:24:55 dist-test-slave-b357 krb5kdc[21333](info): commencing operation
krb5kdc: starting...
W20260621 05:24:56.719439 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:24:56 dist-test-slave-b357 krb5kdc[21333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:24:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:24:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:24:56.837880 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:34369
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:34151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:34369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:24:56.919528 21346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:56.919740 21346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:56.919776 21346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:56.920867 21346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:24:56.920926 21346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:56.920951 21346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:56.920981 21346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:24:56.921010 21346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:24:56.922410 21346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:34151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:34369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:34369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:56.922658 21346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:56.922869 21346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:56.927562 21352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:56.927670 21346 server_base.cc:1034] running on GCE node
W20260621 05:24:56.927623 21351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:56.927755 21354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:56.927991 21346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:56.928268 21346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:56.929442 21346 hybrid_clock.cc:648] HybridClock initialized: now 1782019496929428 us; error 32 us; skew 500 ppm
Jun 21 05:24:56 dist-test-slave-b357 krb5kdc[21333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019496, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:56.932076 21346 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:24:56.932528 21346 webserver.cc:460] Webserver started at http://127.2.32.62:43735/ using document root <none> and password file <none>
I20260621 05:24:56.932744 21346 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:56.932806 21346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:56.932933 21346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:56.934468 21346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "4596da1f85af4a538b0432406124e50c"
format_stamp: "Formatted at 2026-06-21 05:24:56 on dist-test-slave-b357"
server_key: "c418ec0285087a59c834afe77a9561f2"
server_key_iv: "06c2a75e4379a2ea13af3f3132257f81"
server_key_version: "encryptionkey@0"
I20260621 05:24:56.935094 21346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "4596da1f85af4a538b0432406124e50c"
format_stamp: "Formatted at 2026-06-21 05:24:56 on dist-test-slave-b357"
server_key: "c418ec0285087a59c834afe77a9561f2"
server_key_iv: "06c2a75e4379a2ea13af3f3132257f81"
server_key_version: "encryptionkey@0"
I20260621 05:24:56.936916 21346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:56.937814 21361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:56.938042 21346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:56.938153 21346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "4596da1f85af4a538b0432406124e50c"
format_stamp: "Formatted at 2026-06-21 05:24:56 on dist-test-slave-b357"
server_key: "c418ec0285087a59c834afe77a9561f2"
server_key_iv: "06c2a75e4379a2ea13af3f3132257f81"
server_key_version: "encryptionkey@0"
I20260621 05:24:56.938259 21346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:56.949699 21346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:56.952037 21346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:56.952193 21346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:56.956485 21346 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:34369
I20260621 05:24:56.956509 21413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:34369 every 8 connection(s)
I20260621 05:24:56.956838 21346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:24:56.957939 21414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:56.961261 21414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c: Bootstrap starting.
I20260621 05:24:56.962040 21414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:56.962443 21414 log.cc:824] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:56.963384 21414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c: No bootstrap required, opened a new log
I20260621 05:24:56.963696 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 21346
I20260621 05:24:56.963832 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:24:56.964035 2176 external_mini_cluster.cc:1428] Setting key ee32c628af225073e21e85cd50bf4bd8
I20260621 05:24:56.965754 21414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4596da1f85af4a538b0432406124e50c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34369 } }
I20260621 05:24:56.965929 21414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:56.965974 21414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4596da1f85af4a538b0432406124e50c, State: Initialized, Role: FOLLOWER
I20260621 05:24:56.966104 21414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4596da1f85af4a538b0432406124e50c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34369 } }
I20260621 05:24:56.966194 21414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:24:56.966239 21414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:24:56.966288 21414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:56.967379 21414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4596da1f85af4a538b0432406124e50c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34369 } }
I20260621 05:24:56.967491 21414 leader_election.cc:302] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4596da1f85af4a538b0432406124e50c; no voters:
I20260621 05:24:56.967687 21414 leader_election.cc:288] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:24:56.967708 21419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:24:56.967851 21419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c [term 1 LEADER]: Becoming Leader. State: Replica: 4596da1f85af4a538b0432406124e50c, State: Running, Role: LEADER
I20260621 05:24:56.967940 21419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4596da1f85af4a538b0432406124e50c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34369 } }
I20260621 05:24:56.967996 21414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:24:56.968297 21419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4596da1f85af4a538b0432406124e50c. Latest consensus state: current_term: 1 leader_uuid: "4596da1f85af4a538b0432406124e50c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4596da1f85af4a538b0432406124e50c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34369 } } }
I20260621 05:24:56.968395 21419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c [sys.catalog]: This master's current role is: LEADER
I20260621 05:24:56.968313 21420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4596da1f85af4a538b0432406124e50c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4596da1f85af4a538b0432406124e50c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34369 } } }
I20260621 05:24:56.968631 21420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c [sys.catalog]: This master's current role is: LEADER
I20260621 05:24:56.969561 21425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:24:56.970418 21425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:24:56.972210 21425 catalog_manager.cc:1347] Generated new cluster ID: 48c059937e02455b8334937328cd7331
I20260621 05:24:56.972265 21425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:24:56 dist-test-slave-b357 krb5kdc[21333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:24:56.977202 21418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:56.964662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58042 (local address 127.2.32.62:34369)
0621 05:24:56.964880 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:56.964892 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:56.964930 (+ 38us) server_negotiation.cc:407] Connection header received
0621 05:24:56.965741 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:56.965750 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:56.965815 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:56.966010 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:56.966939 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:56.968302 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:56.969364 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:56.970015 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:56.970148 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:56.973258 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:56.973263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:56.973263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:56.973277 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:56.975318 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:56.975685 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:56.975688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:56.975688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:56.975806 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:56.975976 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:56.975977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:56.975977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:56.976283 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:56.976421 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:56.976847 (+ 426us) server_negotiation.cc:299] Negotiation successful
0621 05:24:56.976980 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":117,"threads_started":1}
I20260621 05:24:56.982717 21425 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:24:56.983270 21425 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:24:56.988469 21425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c: Generated new TSK 0
I20260621 05:24:56.988634 21425 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.32.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
Entry for principal kudu/127.2.32.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab.
I20260621 05:24:57.015878 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:34369
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:34151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:24:57.094870 21440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:57.095057 21440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:57.095080 21440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:57.096247 21440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:57.096297 21440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:57.096324 21440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:24:57.097714 21440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:34151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.1.keytab
--principal=kudu/127.2.32.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:34369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:57.097939 21440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:57.098176 21440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:57.103273 21446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:57.103410 21448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:57.103336 21445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:57.103305 21440 server_base.cc:1034] running on GCE node
I20260621 05:24:57.103928 21440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:57.104194 21440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:57.105382 21440 hybrid_clock.cc:648] HybridClock initialized: now 1782019497105390 us; error 74 us; skew 500 ppm
Jun 21 05:24:57 dist-test-slave-b357 krb5kdc[21333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019497, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:57.108254 21440 init.cc:377] Logged in from keytab as kudu/127.2.32.1@KRBTEST.COM (short username kudu)
I20260621 05:24:57.108661 21440 webserver.cc:460] Webserver started at http://127.2.32.1:42195/ using document root <none> and password file <none>
I20260621 05:24:57.108888 21440 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:57.108951 21440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:57.109092 21440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:57.110242 21440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "880044f6910143409111b0adaa58e6bd"
format_stamp: "Formatted at 2026-06-21 05:24:57 on dist-test-slave-b357"
server_key: "aea3d5de94e4564486479c6a8ec2cbbe"
server_key_iv: "393ab6e997b280f21a6e663c11b7dd9e"
server_key_version: "encryptionkey@0"
I20260621 05:24:57.110649 21440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "880044f6910143409111b0adaa58e6bd"
format_stamp: "Formatted at 2026-06-21 05:24:57 on dist-test-slave-b357"
server_key: "aea3d5de94e4564486479c6a8ec2cbbe"
server_key_iv: "393ab6e997b280f21a6e663c11b7dd9e"
server_key_version: "encryptionkey@0"
I20260621 05:24:57.112181 21440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:24:57.113023 21455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:57.113298 21440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:24:57.113443 21440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "880044f6910143409111b0adaa58e6bd"
format_stamp: "Formatted at 2026-06-21 05:24:57 on dist-test-slave-b357"
server_key: "aea3d5de94e4564486479c6a8ec2cbbe"
server_key_iv: "393ab6e997b280f21a6e663c11b7dd9e"
server_key_version: "encryptionkey@0"
I20260621 05:24:57.113544 21440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:57.122311 21440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:57.124526 21440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:57.124682 21440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:57.124907 21440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:57.125190 21440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:57.125238 21440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:57.125274 21440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:57.125300 21440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:57.131718 21440 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:37763
I20260621 05:24:57.131881 21568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:37763 every 8 connection(s)
I20260621 05:24:57.132140 21440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
Jun 21 05:24:57 dist-test-slave-b357 krb5kdc[21333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019497, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.1@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:24:57.140201 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 21440
I20260621 05:24:57.140512 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:24:57.140817 2176 external_mini_cluster.cc:1428] Setting key 8489fff4bece7c6eac6db640a4e8e194
I20260621 05:24:57.142583 21418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.133020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:59289 (local address 127.2.32.62:34369)
0621 05:24:57.133071 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:57.133073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:57.133224 (+ 151us) server_negotiation.cc:407] Connection header received
0621 05:24:57.133748 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:57.133750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:57.133766 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:57.133827 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:57.134789 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.135405 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.136345 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.136551 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.136617 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.138631 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:57.138633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:57.138633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:57.138642 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:57.140100 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:57.141048 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:57.141050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:57.141050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:57.141149 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:57.141702 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:57.141703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:57.141703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:57.141872 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:57.141979 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:57.142378 (+ 399us) server_negotiation.cc:299] Negotiation successful
0621 05:24:57.142503 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:24:57.142834 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.132966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:34369 (local address 127.2.32.1:59289)
0621 05:24:57.133114 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:57.133153 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:57.133608 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:57.133921 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:57.133924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:57.134480 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:57.134660 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:57.134664 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.135485 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.135486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:57.136258 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:57.136260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.136635 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.136636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:57.136730 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.137213 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:24:57.137221 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:57.138530 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:57.140328 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:57.140348 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:57.140352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:57.140857 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:57.141428 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:57.141429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:57.141430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:57.141601 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:57.142001 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:57.142009 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:57.142189 (+ 180us) client_negotiation.cc:769] Sending connection context
0621 05:24:57.142352 (+ 163us) client_negotiation.cc:240] Negotiation successful
0621 05:24:57.142545 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260621 05:24:57.143682 21569 heartbeater.cc:347] Connected to a master server at 127.2.32.62:34369
I20260621 05:24:57.143798 21569 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:57.143973 21569 heartbeater.cc:511] Master 127.2.32.62:34369 requested a full tablet report, sending...
I20260621 05:24:57.144522 21378 ts_manager.cc:194] Registered new tserver with Master: 880044f6910143409111b0adaa58e6bd (127.2.32.1:37763)
I20260621 05:24:57.145430 21378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.1@KRBTEST.COM'} at 127.2.32.1:59289
WARNING: no policy specified for kudu/127.2.32.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
Entry for principal kudu/127.2.32.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab.
I20260621 05:24:57.167001 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:34369
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:34151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:24:57.248410 21574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:57.248610 21574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:57.248633 21574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:57.249756 21574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:57.249804 21574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:57.249831 21574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:24:57.251226 21574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:34151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.2.keytab
--principal=kudu/127.2.32.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:34369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:57.251457 21574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:57.251679 21574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:57.257274 21582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:57.257266 21580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:57.257265 21579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:57.257637 21574 server_base.cc:1034] running on GCE node
I20260621 05:24:57.257798 21574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:57.258020 21574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:57.259215 21574 hybrid_clock.cc:648] HybridClock initialized: now 1782019497259191 us; error 30 us; skew 500 ppm
Jun 21 05:24:57 dist-test-slave-b357 krb5kdc[21333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019497, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:57.261704 21574 init.cc:377] Logged in from keytab as kudu/127.2.32.2@KRBTEST.COM (short username kudu)
I20260621 05:24:57.262112 21574 webserver.cc:460] Webserver started at http://127.2.32.2:44567/ using document root <none> and password file <none>
I20260621 05:24:57.262341 21574 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:57.262405 21574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:57.262544 21574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:57.263644 21574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "c5b96d6edf6c4cfd96254dd8571e3b71"
format_stamp: "Formatted at 2026-06-21 05:24:57 on dist-test-slave-b357"
server_key: "b8fb85e50b02db30f4b29847957642ab"
server_key_iv: "0bc0936e08591b5fd782e9f3fb3a7660"
server_key_version: "encryptionkey@0"
I20260621 05:24:57.264000 21574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "c5b96d6edf6c4cfd96254dd8571e3b71"
format_stamp: "Formatted at 2026-06-21 05:24:57 on dist-test-slave-b357"
server_key: "b8fb85e50b02db30f4b29847957642ab"
server_key_iv: "0bc0936e08591b5fd782e9f3fb3a7660"
server_key_version: "encryptionkey@0"
I20260621 05:24:57.265278 21574 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:24:57.266031 21589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:57.266371 21574 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:57.266557 21574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "c5b96d6edf6c4cfd96254dd8571e3b71"
format_stamp: "Formatted at 2026-06-21 05:24:57 on dist-test-slave-b357"
server_key: "b8fb85e50b02db30f4b29847957642ab"
server_key_iv: "0bc0936e08591b5fd782e9f3fb3a7660"
server_key_version: "encryptionkey@0"
I20260621 05:24:57.266688 21574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:57.280726 21574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:57.282923 21574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:57.283073 21574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:57.283315 21574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:57.283617 21574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:57.283666 21574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:57.283705 21574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:57.283732 21574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:57.289840 21574 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:46663
I20260621 05:24:57.289909 21702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:46663 every 8 connection(s)
I20260621 05:24:57.290184 21574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:24:57.292320 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 21574
I20260621 05:24:57.292474 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:24:57.292856 2176 external_mini_cluster.cc:1428] Setting key 92d1afcf2128f11ade98b26dbf5c6881
Jun 21 05:24:57 dist-test-slave-b357 krb5kdc[21333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019497, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.2@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.32.3@KRBTEST.COM; defaulting to no policy
I20260621 05:24:57.301793 21418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.291056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:58381 (local address 127.2.32.62:34369)
0621 05:24:57.291181 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:57.291184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:57.291471 (+ 287us) server_negotiation.cc:407] Connection header received
0621 05:24:57.292093 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:57.292095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:57.292113 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:57.292195 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:57.293528 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.294420 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.295757 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.296092 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.296186 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.298745 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:57.298748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:57.298748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:57.298758 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:57.300405 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:57.300862 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:57.300863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:57.300864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:57.300922 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:57.301164 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:57.301165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:57.301166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:57.301274 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:57.301347 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:57.301614 (+ 267us) server_negotiation.cc:299] Negotiation successful
0621 05:24:57.301698 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260621 05:24:57.301983 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.291007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:34369 (local address 127.2.32.2:58381)
0621 05:24:57.291290 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:57.291346 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:57.291928 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:57.292285 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:57.292289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:57.293164 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:57.293434 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:57.293439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.294588 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.294589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:57.295653 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:57.295657 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.296215 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.296216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:57.296341 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.296835 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:24:57.296844 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:57.298635 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:57.300510 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:57.300517 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:57.300519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:57.300763 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:57.300987 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:57.300987 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:57.300988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:57.301098 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:57.301340 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:57.301355 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:57.301488 (+ 133us) client_negotiation.cc:769] Sending connection context
0621 05:24:57.301594 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 05:24:57.301760 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":169,"threads_started":1}
I20260621 05:24:57.302495 21703 heartbeater.cc:347] Connected to a master server at 127.2.32.62:34369
I20260621 05:24:57.302600 21703 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:57.302783 21703 heartbeater.cc:511] Master 127.2.32.62:34369 requested a full tablet report, sending...
I20260621 05:24:57.303210 21378 ts_manager.cc:194] Registered new tserver with Master: c5b96d6edf6c4cfd96254dd8571e3b71 (127.2.32.2:46663)
I20260621 05:24:57.303565 21378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.2@KRBTEST.COM'} at 127.2.32.2:58381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
Entry for principal kudu/127.2.32.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab.
I20260621 05:24:57.319280 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:34369
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:34151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:24:57.400241 21708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:57.400440 21708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:57.400465 21708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:57.401504 21708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:57.401538 21708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:57.401566 21708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:24:57.402927 21708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:34151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.3.keytab
--principal=kudu/127.2.32.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:34369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:57.403115 21708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:57.403384 21708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:57.408242 21713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:57.408370 21714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:57.408393 21716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:57.408612 21708 server_base.cc:1034] running on GCE node
I20260621 05:24:57.408823 21708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:57.409062 21708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:57.410266 21708 hybrid_clock.cc:648] HybridClock initialized: now 1782019497410245 us; error 39 us; skew 500 ppm
Jun 21 05:24:57 dist-test-slave-b357 krb5kdc[21333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019497, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:57.413232 21708 init.cc:377] Logged in from keytab as kudu/127.2.32.3@KRBTEST.COM (short username kudu)
I20260621 05:24:57.413969 21708 webserver.cc:460] Webserver started at http://127.2.32.3:35265/ using document root <none> and password file <none>
I20260621 05:24:57.414263 21708 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:57.414336 21708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:57.414505 21708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:57.415871 21708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "ac4887487c9048508a16ec6f604e5376"
format_stamp: "Formatted at 2026-06-21 05:24:57 on dist-test-slave-b357"
server_key: "bf116495d77854d2f5ad66d5bc7e0e1b"
server_key_iv: "f95a33be01012a7243da7078d11746d5"
server_key_version: "encryptionkey@0"
I20260621 05:24:57.416309 21708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "ac4887487c9048508a16ec6f604e5376"
format_stamp: "Formatted at 2026-06-21 05:24:57 on dist-test-slave-b357"
server_key: "bf116495d77854d2f5ad66d5bc7e0e1b"
server_key_iv: "f95a33be01012a7243da7078d11746d5"
server_key_version: "encryptionkey@0"
I20260621 05:24:57.417994 21708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:24:57.419023 21723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:57.419268 21708 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:57.419390 21708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "ac4887487c9048508a16ec6f604e5376"
format_stamp: "Formatted at 2026-06-21 05:24:57 on dist-test-slave-b357"
server_key: "bf116495d77854d2f5ad66d5bc7e0e1b"
server_key_iv: "f95a33be01012a7243da7078d11746d5"
server_key_version: "encryptionkey@0"
I20260621 05:24:57.419493 21708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:57.432214 21708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:57.434856 21708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:57.435062 21708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:57.435374 21708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:57.435753 21708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:57.435808 21708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:57.435853 21708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:57.435878 21708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:57.442730 21708 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:38073
I20260621 05:24:57.442770 21836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:38073 every 8 connection(s)
I20260621 05:24:57.443096 21708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:24:57.444059 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 21708
I20260621 05:24:57.444191 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:24:57.444429 2176 external_mini_cluster.cc:1428] Setting key 953b4ebffd527ef8df874cff96542431
Jun 21 05:24:57 dist-test-slave-b357 krb5kdc[21333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019497, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.3@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:24:57.453583 21418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.444103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:47693 (local address 127.2.32.62:34369)
0621 05:24:57.444163 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:57.444166 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:57.444505 (+ 339us) server_negotiation.cc:407] Connection header received
0621 05:24:57.445085 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:57.445086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:57.445099 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:57.445145 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:57.446186 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.446694 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.447713 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.447907 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.447988 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.450478 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:57.450480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:57.450480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:57.450490 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:57.452046 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:57.452563 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:57.452564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:57.452564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:57.452632 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:57.452899 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:57.452900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:57.452901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:57.453058 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:57.453137 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:57.453448 (+ 311us) server_negotiation.cc:299] Negotiation successful
0621 05:24:57.453517 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 05:24:57.453742 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.444082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:34369 (local address 127.2.32.3:47693)
0621 05:24:57.444279 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:57.444328 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:57.444982 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:57.445211 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:57.445215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:57.445903 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:57.446084 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:57.446088 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.446781 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.446782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:57.447626 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:57.447629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.448005 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.448006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:57.448118 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.448678 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:24:57.448686 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:57.450326 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:57.452193 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:57.452201 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:57.452203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:57.452460 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:57.452717 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:24:57.452718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:24:57.452719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:24:57.452828 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:24:57.453149 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:57.453155 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:57.453308 (+ 153us) client_negotiation.cc:769] Sending connection context
0621 05:24:57.453413 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 05:24:57.453534 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":101,"threads_started":1}
I20260621 05:24:57.454205 21837 heartbeater.cc:347] Connected to a master server at 127.2.32.62:34369
I20260621 05:24:57.454293 21837 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:57.454432 21837 heartbeater.cc:511] Master 127.2.32.62:34369 requested a full tablet report, sending...
I20260621 05:24:57.454768 21378 ts_manager.cc:194] Registered new tserver with Master: ac4887487c9048508a16ec6f604e5376 (127.2.32.3:38073)
I20260621 05:24:57.455102 21378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.32.3@KRBTEST.COM'} at 127.2.32.3:47693
I20260621 05:24:57.456668 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:24:57.464732 21418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.458211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58048 (local address 127.2.32.62:34369)
0621 05:24:57.458282 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:57.458284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:57.458296 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:24:57.458331 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:57.458332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:57.458342 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:57.458418 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:57.458954 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.459595 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.460348 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.460638 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.460723 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.462149 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:57.462150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:57.462150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:57.462157 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:57.463581 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:57.463941 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:57.463942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:57.463943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:57.463997 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:57.464197 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:57.464198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:57.464198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:57.464423 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:57.464503 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:57.464604 (+ 101us) server_negotiation.cc:299] Negotiation successful
0621 05:24:57.464672 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 05:24:57.466341 21378 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:24:57.468029 21378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:24:57.474936 21847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.470466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.2.32.2:46663)
0621 05:24:57.470866 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:57.470872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:57.470953 (+ 81us) server_negotiation.cc:407] Connection header received
0621 05:24:57.471345 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:57.471348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:57.471367 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:57.471491 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:57.471630 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.472458 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.473752 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.474481 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.474571 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.474630 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:57.474790 (+ 160us) server_negotiation.cc:299] Negotiation successful
0621 05:24:57.474846 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":84,"threads_started":1}
I20260621 05:24:57.474992 21846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.470376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:37763 (local address 127.0.0.1:40988)
0621 05:24:57.470538 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:57.470593 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:57.470683 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:57.471029 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:57.471032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:57.471035 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:57.471103 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:57.471104 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.472486 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.472488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:57.473878 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:57.473881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.474720 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.474722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:57.474811 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.474814 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:24:57.474866 (+ 52us) client_negotiation.cc:240] Negotiation successful
0621 05:24:57.474924 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260621 05:24:57.475001 21849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.470589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40988 (local address 127.2.32.1:37763)
0621 05:24:57.470820 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:57.470826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:57.470838 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:24:57.470886 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:57.470889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:57.470908 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:57.471203 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:57.471297 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.472355 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.473965 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.474657 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.474718 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.474765 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:57.474883 (+ 118us) server_negotiation.cc:299] Negotiation successful
0621 05:24:57.474918 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":111,"threads_started":1}
I20260621 05:24:57.475230 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.470378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:46663 (local address 127.0.0.1:53188)
0621 05:24:57.470874 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:57.470886 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:57.470988 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:57.471489 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:57.471490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:57.471493 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:57.471562 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:57.471565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.472678 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.472680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:57.473662 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:57.473664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.474578 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.474579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:57.474654 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.474667 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 05:24:57.475060 (+ 393us) client_negotiation.cc:240] Negotiation successful
0621 05:24:57.475105 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":66,"threads_started":1}
I20260621 05:24:57.475340 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.470584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:38073 (local address 127.0.0.1:40318)
0621 05:24:57.471158 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:57.471169 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:57.471235 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:57.471646 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:57.471647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:57.471648 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:57.471694 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:57.471696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.473129 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.473130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:57.474151 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:57.474153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.474977 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.474978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:57.475190 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.475194 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 05:24:57.475237 (+ 43us) client_negotiation.cc:240] Negotiation successful
0621 05:24:57.475280 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":56,"threads_started":1}
I20260621 05:24:57.475575 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.470700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40318 (local address 127.2.32.3:38073)
0621 05:24:57.470995 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:57.470999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:57.471239 (+ 240us) server_negotiation.cc:407] Connection header received
0621 05:24:57.471357 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:57.471360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:57.471376 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:57.471490 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:57.471781 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.472836 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.474229 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.474831 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.474893 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.474941 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:57.475440 (+ 499us) server_negotiation.cc:299] Negotiation successful
0621 05:24:57.475493 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":101,"threads_started":1}
I20260621 05:24:57.475873 21503 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4f557b9fcb4425aee4c8526ff3ea91 (DEFAULT_TABLE table=test-table [id=e80311b61f2e45af87f83260e103cc32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:57.475914 21637 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4f557b9fcb4425aee4c8526ff3ea91 (DEFAULT_TABLE table=test-table [id=e80311b61f2e45af87f83260e103cc32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:57.476073 21771 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4f557b9fcb4425aee4c8526ff3ea91 (DEFAULT_TABLE table=test-table [id=e80311b61f2e45af87f83260e103cc32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:57.476248 21637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4f557b9fcb4425aee4c8526ff3ea91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:57.476235 21503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4f557b9fcb4425aee4c8526ff3ea91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:57.476336 21771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4f557b9fcb4425aee4c8526ff3ea91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:57.478957 21851 tablet_bootstrap.cc:492] T 0e4f557b9fcb4425aee4c8526ff3ea91 P 880044f6910143409111b0adaa58e6bd: Bootstrap starting.
I20260621 05:24:57.479635 21851 tablet_bootstrap.cc:654] T 0e4f557b9fcb4425aee4c8526ff3ea91 P 880044f6910143409111b0adaa58e6bd: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:57.480218 21853 tablet_bootstrap.cc:492] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376: Bootstrap starting.
I20260621 05:24:57.480247 21851 log.cc:824] T 0e4f557b9fcb4425aee4c8526ff3ea91 P 880044f6910143409111b0adaa58e6bd: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:57.481236 21853 tablet_bootstrap.cc:654] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:57.481274 21851 tablet_bootstrap.cc:492] T 0e4f557b9fcb4425aee4c8526ff3ea91 P 880044f6910143409111b0adaa58e6bd: No bootstrap required, opened a new log
I20260621 05:24:57.481348 21851 ts_tablet_manager.cc:1397] T 0e4f557b9fcb4425aee4c8526ff3ea91 P 880044f6910143409111b0adaa58e6bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:24:57.481977 21853 log.cc:824] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:57.482355 21852 tablet_bootstrap.cc:492] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71: Bootstrap starting.
I20260621 05:24:57.483031 21852 tablet_bootstrap.cc:654] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:57.483634 21853 tablet_bootstrap.cc:492] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376: No bootstrap required, opened a new log
I20260621 05:24:57.483664 21852 log.cc:824] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:57.483740 21853 ts_tablet_manager.cc:1397] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260621 05:24:57.483783 21851 raft_consensus.cc:348] T 0e4f557b9fcb4425aee4c8526ff3ea91 P 880044f6910143409111b0adaa58e6bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880044f6910143409111b0adaa58e6bd" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37763 } } peers { permanent_uuid: "c5b96d6edf6c4cfd96254dd8571e3b71" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46663 } } peers { permanent_uuid: "ac4887487c9048508a16ec6f604e5376" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38073 } }
I20260621 05:24:57.484007 21851 raft_consensus.cc:374] T 0e4f557b9fcb4425aee4c8526ff3ea91 P 880044f6910143409111b0adaa58e6bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:57.484056 21851 raft_consensus.cc:729] T 0e4f557b9fcb4425aee4c8526ff3ea91 P 880044f6910143409111b0adaa58e6bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 880044f6910143409111b0adaa58e6bd, State: Initialized, Role: FOLLOWER
I20260621 05:24:57.484190 21851 consensus_queue.cc:260] T 0e4f557b9fcb4425aee4c8526ff3ea91 P 880044f6910143409111b0adaa58e6bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880044f6910143409111b0adaa58e6bd" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37763 } } peers { permanent_uuid: "c5b96d6edf6c4cfd96254dd8571e3b71" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46663 } } peers { permanent_uuid: "ac4887487c9048508a16ec6f604e5376" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38073 } }
I20260621 05:24:57.484478 21851 ts_tablet_manager.cc:1428] T 0e4f557b9fcb4425aee4c8526ff3ea91 P 880044f6910143409111b0adaa58e6bd: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260621 05:24:57.484617 21569 heartbeater.cc:503] Master 127.2.32.62:34369 was elected leader, sending a full tablet report...
I20260621 05:24:57.484571 21852 tablet_bootstrap.cc:492] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71: No bootstrap required, opened a new log
I20260621 05:24:57.484917 21852 ts_tablet_manager.cc:1397] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 05:24:57.486025 21853 raft_consensus.cc:348] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880044f6910143409111b0adaa58e6bd" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37763 } } peers { permanent_uuid: "c5b96d6edf6c4cfd96254dd8571e3b71" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46663 } } peers { permanent_uuid: "ac4887487c9048508a16ec6f604e5376" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38073 } }
I20260621 05:24:57.486284 21853 raft_consensus.cc:374] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:57.486356 21853 raft_consensus.cc:729] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac4887487c9048508a16ec6f604e5376, State: Initialized, Role: FOLLOWER
I20260621 05:24:57.486434 21852 raft_consensus.cc:348] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880044f6910143409111b0adaa58e6bd" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37763 } } peers { permanent_uuid: "c5b96d6edf6c4cfd96254dd8571e3b71" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46663 } } peers { permanent_uuid: "ac4887487c9048508a16ec6f604e5376" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38073 } }
I20260621 05:24:57.486517 21853 consensus_queue.cc:260] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880044f6910143409111b0adaa58e6bd" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37763 } } peers { permanent_uuid: "c5b96d6edf6c4cfd96254dd8571e3b71" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46663 } } peers { permanent_uuid: "ac4887487c9048508a16ec6f604e5376" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38073 } }
I20260621 05:24:57.486616 21852 raft_consensus.cc:374] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:57.486644 21852 raft_consensus.cc:729] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b96d6edf6c4cfd96254dd8571e3b71, State: Initialized, Role: FOLLOWER
I20260621 05:24:57.486752 21852 consensus_queue.cc:260] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880044f6910143409111b0adaa58e6bd" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37763 } } peers { permanent_uuid: "c5b96d6edf6c4cfd96254dd8571e3b71" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46663 } } peers { permanent_uuid: "ac4887487c9048508a16ec6f604e5376" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38073 } }
I20260621 05:24:57.486891 21853 ts_tablet_manager.cc:1428] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260621 05:24:57.487027 21703 heartbeater.cc:503] Master 127.2.32.62:34369 was elected leader, sending a full tablet report...
I20260621 05:24:57.487063 21837 heartbeater.cc:503] Master 127.2.32.62:34369 was elected leader, sending a full tablet report...
I20260621 05:24:57.487017 21852 ts_tablet_manager.cc:1428] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:24:57.489872 21858 raft_consensus.cc:482] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:24:57.490006 21858 raft_consensus.cc:504] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880044f6910143409111b0adaa58e6bd" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37763 } } peers { permanent_uuid: "c5b96d6edf6c4cfd96254dd8571e3b71" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46663 } } peers { permanent_uuid: "ac4887487c9048508a16ec6f604e5376" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38073 } }
I20260621 05:24:57.490384 21858 leader_election.cc:288] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 880044f6910143409111b0adaa58e6bd (127.2.32.1:37763), c5b96d6edf6c4cfd96254dd8571e3b71 (127.2.32.2:46663)
I20260621 05:24:57.494424 21849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.490804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:50539 (local address 127.2.32.1:37763)
0621 05:24:57.490867 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:57.490869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:57.491065 (+ 196us) server_negotiation.cc:407] Connection header received
0621 05:24:57.491122 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:57.491123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:57.491176 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:57.491241 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:57.491519 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.491992 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.493482 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.494044 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.494132 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.494142 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:57.494321 (+ 179us) server_negotiation.cc:299] Negotiation successful
0621 05:24:57.494358 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260621 05:24:57.494418 21847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.490735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:56925 (local address 127.2.32.2:46663)
0621 05:24:57.490789 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:57.490792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:57.490802 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:24:57.490827 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:57.490827 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:57.490838 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:57.490917 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:57.491116 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.491951 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.493482 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.494064 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.494132 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.494143 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:57.494321 (+ 178us) server_negotiation.cc:299] Negotiation successful
0621 05:24:57.494358 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 05:24:57.494444 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.490616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:46663 (local address 127.2.32.3:56925)
0621 05:24:57.490681 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:57.490697 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:57.490743 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:57.490938 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:57.490939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:57.490940 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:57.491012 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:57.491014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.492032 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.492033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:57.493371 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:57.493373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.494147 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.494148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:57.494240 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.494242 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:24:57.494307 (+ 65us) client_negotiation.cc:240] Negotiation successful
0621 05:24:57.494358 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":67}
I20260621 05:24:57.494457 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.490636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:37763 (local address 127.2.32.3:50539)
0621 05:24:57.490991 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:57.491002 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:57.491054 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:57.491270 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:57.491271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:57.491273 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:57.491347 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:57.491349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.492091 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.492092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:57.493370 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:57.493373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.494147 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.494148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:57.494235 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.494240 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:24:57.494307 (+ 67us) client_negotiation.cc:240] Negotiation successful
0621 05:24:57.494354 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":44,"thread_start_us":122,"threads_started":1}
I20260621 05:24:57.494750 21657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4f557b9fcb4425aee4c8526ff3ea91" candidate_uuid: "ac4887487c9048508a16ec6f604e5376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b96d6edf6c4cfd96254dd8571e3b71" is_pre_election: true
I20260621 05:24:57.494817 21523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4f557b9fcb4425aee4c8526ff3ea91" candidate_uuid: "ac4887487c9048508a16ec6f604e5376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "880044f6910143409111b0adaa58e6bd" is_pre_election: true
I20260621 05:24:57.494910 21523 raft_consensus.cc:2454] T 0e4f557b9fcb4425aee4c8526ff3ea91 P 880044f6910143409111b0adaa58e6bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac4887487c9048508a16ec6f604e5376 in term 0.
I20260621 05:24:57.494903 21657 raft_consensus.cc:2454] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac4887487c9048508a16ec6f604e5376 in term 0.
I20260621 05:24:57.495112 21726 leader_election.cc:302] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 880044f6910143409111b0adaa58e6bd, ac4887487c9048508a16ec6f604e5376; no voters:
I20260621 05:24:57.495283 21858 raft_consensus.cc:2781] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:24:57.495334 21858 raft_consensus.cc:482] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:24:57.495355 21858 raft_consensus.cc:3037] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:57.496023 21858 raft_consensus.cc:504] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880044f6910143409111b0adaa58e6bd" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37763 } } peers { permanent_uuid: "c5b96d6edf6c4cfd96254dd8571e3b71" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46663 } } peers { permanent_uuid: "ac4887487c9048508a16ec6f604e5376" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38073 } }
I20260621 05:24:57.496284 21858 leader_election.cc:288] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [CANDIDATE]: Term 1 election: Requested vote from peers 880044f6910143409111b0adaa58e6bd (127.2.32.1:37763), c5b96d6edf6c4cfd96254dd8571e3b71 (127.2.32.2:46663)
I20260621 05:24:57.496505 21523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4f557b9fcb4425aee4c8526ff3ea91" candidate_uuid: "ac4887487c9048508a16ec6f604e5376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "880044f6910143409111b0adaa58e6bd"
I20260621 05:24:57.496595 21657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4f557b9fcb4425aee4c8526ff3ea91" candidate_uuid: "ac4887487c9048508a16ec6f604e5376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b96d6edf6c4cfd96254dd8571e3b71"
I20260621 05:24:57.496707 21523 raft_consensus.cc:3037] T 0e4f557b9fcb4425aee4c8526ff3ea91 P 880044f6910143409111b0adaa58e6bd [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:57.496735 21657 raft_consensus.cc:3037] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:57.497660 21657 raft_consensus.cc:2454] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac4887487c9048508a16ec6f604e5376 in term 1.
I20260621 05:24:57.497660 21523 raft_consensus.cc:2454] T 0e4f557b9fcb4425aee4c8526ff3ea91 P 880044f6910143409111b0adaa58e6bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac4887487c9048508a16ec6f604e5376 in term 1.
I20260621 05:24:57.497869 21726 leader_election.cc:302] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 880044f6910143409111b0adaa58e6bd, ac4887487c9048508a16ec6f604e5376; no voters:
I20260621 05:24:57.497999 21858 raft_consensus.cc:2781] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:24:57.498210 21858 raft_consensus.cc:686] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [term 1 LEADER]: Becoming Leader. State: Replica: ac4887487c9048508a16ec6f604e5376, State: Running, Role: LEADER
I20260621 05:24:57.498317 21858 consensus_queue.cc:237] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880044f6910143409111b0adaa58e6bd" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37763 } } peers { permanent_uuid: "c5b96d6edf6c4cfd96254dd8571e3b71" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46663 } } peers { permanent_uuid: "ac4887487c9048508a16ec6f604e5376" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38073 } }
I20260621 05:24:57.499245 21377 catalog_manager.cc:5495] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac4887487c9048508a16ec6f604e5376 (127.2.32.3). New cstate: current_term: 1 leader_uuid: "ac4887487c9048508a16ec6f604e5376" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880044f6910143409111b0adaa58e6bd" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5b96d6edf6c4cfd96254dd8571e3b71" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac4887487c9048508a16ec6f604e5376" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 38073 } health_report { overall_health: HEALTHY } } }
I20260621 05:24:57.512179 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.508993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.2.32.3:38073)
0621 05:24:57.509037 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:57.509039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:57.509105 (+ 66us) server_negotiation.cc:407] Connection header received
0621 05:24:57.509165 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:57.509166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:57.509181 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:57.509280 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:24:57.509519 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.509981 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.511002 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.511280 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.511369 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.511559 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:24:57.511853 (+ 294us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:24:57.512005 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:57.512028 (+ 23us) server_negotiation.cc:299] Negotiation successful
0621 05:24:57.512055 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:24:57.514286 21523 raft_consensus.cc:1261] T 0e4f557b9fcb4425aee4c8526ff3ea91 P 880044f6910143409111b0adaa58e6bd [term 1 FOLLOWER]: Refusing update from remote peer ac4887487c9048508a16ec6f604e5376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:24:57.514286 21657 raft_consensus.cc:1261] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71 [term 1 FOLLOWER]: Refusing update from remote peer ac4887487c9048508a16ec6f604e5376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:24:57.514648 21858 consensus_queue.cc:1035] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5b96d6edf6c4cfd96254dd8571e3b71" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 46663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:24:57.514793 21861 consensus_queue.cc:1035] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "880044f6910143409111b0adaa58e6bd" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 37763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:24:57.517519 21864 mvcc.cc:205] Tried to move back new op lower bound from 7299151861815713792 to 7299151861753942016. Current Snapshot: MvccSnapshot[applied={T|T < 7299151861815713792}]
I20260621 05:24:57.518040 21865 mvcc.cc:205] Tried to move back new op lower bound from 7299151861815713792 to 7299151861753942016. Current Snapshot: MvccSnapshot[applied={T|T < 7299151861815713792}]
I20260621 05:24:57.518473 21866 mvcc.cc:205] Tried to move back new op lower bound from 7299151861815713792 to 7299151861753942016. Current Snapshot: MvccSnapshot[applied={T|T < 7299151861815713792}]
I20260621 05:24:57.520161 21376 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:24:57.520246 21376 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:24:57.520972 21376 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4596da1f85af4a538b0432406124e50c: Sending DeleteTablet for 3 replicas of tablet 0e4f557b9fcb4425aee4c8526ff3ea91
I20260621 05:24:57.521270 21637 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e4f557b9fcb4425aee4c8526ff3ea91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:24:57 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:53188
I20260621 05:24:57.521296 21771 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e4f557b9fcb4425aee4c8526ff3ea91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:24:57 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:40318
I20260621 05:24:57.521446 21503 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e4f557b9fcb4425aee4c8526ff3ea91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:24:57 UTC) from {username='kudu', principal='kudu/127.2.32.62@KRBTEST.COM'} at 127.0.0.1:40988
I20260621 05:24:57.521621 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 21440
I20260621 05:24:57.522810 21875 tablet_replica.cc:330] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376: stopping tablet replica
I20260621 05:24:57.522914 21875 raft_consensus.cc:2229] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [term 1 LEADER]: Raft consensus shutting down.
I20260621 05:24:57.523006 21875 raft_consensus.cc:2258] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:24:57.523102 21874 tablet_replica.cc:330] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71: stopping tablet replica
I20260621 05:24:57.523204 21874 raft_consensus.cc:2229] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:24:57.523298 21874 raft_consensus.cc:2258] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:24:57.523504 21875 ts_tablet_manager.cc:1905] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:24:57.525111 21875 ts_tablet_manager.cc:1918] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:24:57.525187 21875 log.cc:1196] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/0e4f557b9fcb4425aee4c8526ff3ea91
I20260621 05:24:57.525465 21875 ts_tablet_manager.cc:1939] T 0e4f557b9fcb4425aee4c8526ff3ea91 P ac4887487c9048508a16ec6f604e5376: Deleting consensus metadata
I20260621 05:24:57.525868 21363 catalog_manager.cc:4841] TS ac4887487c9048508a16ec6f604e5376 (127.2.32.3:38073): tablet 0e4f557b9fcb4425aee4c8526ff3ea91 (table test-table [id=e80311b61f2e45af87f83260e103cc32]) successfully deleted
I20260621 05:24:57.526201 21874 ts_tablet_manager.cc:1905] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:24:57.528082 21874 ts_tablet_manager.cc:1918] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:24:57.528155 21874 log.cc:1196] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/0e4f557b9fcb4425aee4c8526ff3ea91
I20260621 05:24:57.528420 21874 ts_tablet_manager.cc:1939] T 0e4f557b9fcb4425aee4c8526ff3ea91 P c5b96d6edf6c4cfd96254dd8571e3b71: Deleting consensus metadata
W20260621 05:24:57.528625 21364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.32.1:37763 (error 108)
I20260621 05:24:57.528776 21363 catalog_manager.cc:4841] TS c5b96d6edf6c4cfd96254dd8571e3b71 (127.2.32.2:46663): tablet 0e4f557b9fcb4425aee4c8526ff3ea91 (table test-table [id=e80311b61f2e45af87f83260e103cc32]) successfully deleted
I20260621 05:24:57.528954 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 21574
I20260621 05:24:57.529714 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.529101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:37763 (local address 127.0.0.1:40990)
0621 05:24:57.529656 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:57.529691 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.1:37763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536}
W20260621 05:24:57.529829 21364 catalog_manager.cc:4584] TS 880044f6910143409111b0adaa58e6bd (127.2.32.1:37763): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0e4f557b9fcb4425aee4c8526ff3ea91: Network error: Client connection negotiation failed: client connection to 127.2.32.1:37763: connect: Connection refused (error 111)
I20260621 05:24:57.534795 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 21708
I20260621 05:24:57.539551 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 21346
2026-06-21T05:24:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1881 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-21T05:24:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:24:57Z Disabled control of system clock
I20260621 05:24:57.570228 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:46071
--webserver_interface=127.2.32.62
--webserver_port=0
--builtin_ntp_servers=127.2.32.20:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:46071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:24:57.649399 21881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:57.649596 21881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:57.649616 21881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:57.650651 21881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:24:57.650682 21881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:57.650695 21881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:57.650707 21881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:24:57.650719 21881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:24:57.651963 21881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:46071
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:46071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:57.652137 21881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:57.652354 21881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:57.657269 21886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:57.657189 21889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:57.657203 21887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:57.657606 21881 server_base.cc:1034] running on GCE node
I20260621 05:24:57.657989 21881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:57.658373 21881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:57.659528 21881 hybrid_clock.cc:648] HybridClock initialized: now 1782019497659490 us; error 57 us; skew 500 ppm
I20260621 05:24:57.661190 21881 webserver.cc:460] Webserver started at http://127.2.32.62:41437/ using document root <none> and password file <none>
I20260621 05:24:57.661515 21881 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:57.661590 21881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:57.661756 21881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:57.663017 21881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "49e60f9ef35e45e59bfc524c2257918f"
format_stamp: "Formatted at 2026-06-21 05:24:57 on dist-test-slave-b357"
server_key: "ef31f34e4e380351fe15e868b8e8f448"
server_key_iv: "fd19552943df3cd5270dc605927aacf5"
server_key_version: "encryptionkey@0"
I20260621 05:24:57.663754 21881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "49e60f9ef35e45e59bfc524c2257918f"
format_stamp: "Formatted at 2026-06-21 05:24:57 on dist-test-slave-b357"
server_key: "ef31f34e4e380351fe15e868b8e8f448"
server_key_iv: "fd19552943df3cd5270dc605927aacf5"
server_key_version: "encryptionkey@0"
I20260621 05:24:57.665488 21881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:24:57.666241 21895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:57.666409 21881 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260621 05:24:57.666520 21881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "49e60f9ef35e45e59bfc524c2257918f"
format_stamp: "Formatted at 2026-06-21 05:24:57 on dist-test-slave-b357"
server_key: "ef31f34e4e380351fe15e868b8e8f448"
server_key_iv: "fd19552943df3cd5270dc605927aacf5"
server_key_version: "encryptionkey@0"
I20260621 05:24:57.666602 21881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:57.679607 21881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:57.679893 21881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:57.680040 21881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:57.683872 21881 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:46071
I20260621 05:24:57.683913 21947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:46071 every 8 connection(s)
I20260621 05:24:57.684247 21881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:24:57.685169 21948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:57.685539 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 21881
I20260621 05:24:57.685683 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:24:57.685887 2176 external_mini_cluster.cc:1428] Setting key c51bd9646412297bd43fc24292c2de62
I20260621 05:24:57.687867 21948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f: Bootstrap starting.
I20260621 05:24:57.688479 21948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:57.688781 21948 log.cc:824] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:57.689455 21948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f: No bootstrap required, opened a new log
I20260621 05:24:57.691217 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.686414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55644 (local address 127.2.32.62:46071)
0621 05:24:57.686589 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:57.686595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:57.686619 (+ 24us) server_negotiation.cc:407] Connection header received
0621 05:24:57.686997 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:57.687003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:57.687054 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:57.687230 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:57.687447 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.688286 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.689282 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.689701 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.689785 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.690518 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:57.690522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:57.690522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:24:57.690683 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:57.690713 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:24:57.690723 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:57.690778 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:57.690891 (+ 113us) server_negotiation.cc:299] Negotiation successful
0621 05:24:57.691001 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260621 05:24:57.691744 21948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e60f9ef35e45e59bfc524c2257918f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46071 } }
I20260621 05:24:57.691919 21948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:57.691970 21948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e60f9ef35e45e59bfc524c2257918f, State: Initialized, Role: FOLLOWER
I20260621 05:24:57.692098 21948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e60f9ef35e45e59bfc524c2257918f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46071 } }
I20260621 05:24:57.692170 21948 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:24:57.692201 21948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:24:57.692265 21948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:57.692746 21948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e60f9ef35e45e59bfc524c2257918f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46071 } }
I20260621 05:24:57.692827 21948 leader_election.cc:302] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49e60f9ef35e45e59bfc524c2257918f; no voters:
I20260621 05:24:57.692992 21948 leader_election.cc:288] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:24:57.693084 21953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:24:57.693235 21953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f [term 1 LEADER]: Becoming Leader. State: Replica: 49e60f9ef35e45e59bfc524c2257918f, State: Running, Role: LEADER
I20260621 05:24:57.693253 21948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:24:57.693341 21953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e60f9ef35e45e59bfc524c2257918f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46071 } }
I20260621 05:24:57.693665 21955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49e60f9ef35e45e59bfc524c2257918f. Latest consensus state: current_term: 1 leader_uuid: "49e60f9ef35e45e59bfc524c2257918f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e60f9ef35e45e59bfc524c2257918f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46071 } } }
I20260621 05:24:57.693655 21954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49e60f9ef35e45e59bfc524c2257918f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49e60f9ef35e45e59bfc524c2257918f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 46071 } } }
I20260621 05:24:57.693751 21955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f [sys.catalog]: This master's current role is: LEADER
I20260621 05:24:57.693778 21954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f [sys.catalog]: This master's current role is: LEADER
I20260621 05:24:57.693943 21960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:24:57.694433 21960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:24:57.696048 21960 catalog_manager.cc:1347] Generated new cluster ID: c587d09f0cf34c3ab54418ebf978a8f7
I20260621 05:24:57.696110 21960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:24:57.703418 21960 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:24:57.703919 21960 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:24:57.712527 21960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f: Generated new TSK 0
I20260621 05:24:57.712707 21960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:24:57.717798 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.1:0
--local_ip_for_outbound_sockets=127.2.32.1
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46071
--builtin_ntp_servers=127.2.32.20:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:24:57.804766 21972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:57.805015 21972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:57.805043 21972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:57.806411 21972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:57.806456 21972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:57.806500 21972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.1
I20260621 05:24:57.808283 21972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.32.1
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.1
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:57.808591 21972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:57.808954 21972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:57.814620 21977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:57.814620 21978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:57.814631 21980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:57.814978 21972 server_base.cc:1034] running on GCE node
I20260621 05:24:57.815215 21972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:57.815486 21972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:57.816668 21972 hybrid_clock.cc:648] HybridClock initialized: now 1782019497816651 us; error 33 us; skew 500 ppm
I20260621 05:24:57.818203 21972 webserver.cc:460] Webserver started at http://127.2.32.1:42105/ using document root <none> and password file <none>
I20260621 05:24:57.818522 21972 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:57.818588 21972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:57.818724 21972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:57.819892 21972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-0/data/instance:
uuid: "b637f6e64db64f88b31f3c0a0f29fb4b"
format_stamp: "Formatted at 2026-06-21 05:24:57 on dist-test-slave-b357"
server_key: "c8f3fcfd0b14a05227856e108cfab90b"
server_key_iv: "b6af5e07bb9c97f397409dbfc5cb431f"
server_key_version: "encryptionkey@0"
I20260621 05:24:57.820251 21972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance:
uuid: "b637f6e64db64f88b31f3c0a0f29fb4b"
format_stamp: "Formatted at 2026-06-21 05:24:57 on dist-test-slave-b357"
server_key: "c8f3fcfd0b14a05227856e108cfab90b"
server_key_iv: "b6af5e07bb9c97f397409dbfc5cb431f"
server_key_version: "encryptionkey@0"
I20260621 05:24:57.821574 21972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:24:57.822432 21986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:57.822752 21972 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:57.822918 21972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-0/wal
uuid: "b637f6e64db64f88b31f3c0a0f29fb4b"
format_stamp: "Formatted at 2026-06-21 05:24:57 on dist-test-slave-b357"
server_key: "c8f3fcfd0b14a05227856e108cfab90b"
server_key_iv: "b6af5e07bb9c97f397409dbfc5cb431f"
server_key_version: "encryptionkey@0"
I20260621 05:24:57.822999 21972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:57.847880 21972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:57.848198 21972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:57.848349 21972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:57.848590 21972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:57.848953 21972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:57.849007 21972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:57.849041 21972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:57.849071 21972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:57.855574 21972 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.1:36387
I20260621 05:24:57.855695 22099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.1:36387 every 8 connection(s)
I20260621 05:24:57.856003 21972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-0/data/info.pb
I20260621 05:24:57.861335 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.856798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:43477 (local address 127.2.32.62:46071)
0621 05:24:57.856873 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:57.856875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:57.857051 (+ 176us) server_negotiation.cc:407] Connection header received
0621 05:24:57.857731 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:57.857732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:57.857750 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:57.857805 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:57.858484 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.858988 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.859968 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.860154 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.860225 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.861035 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:57.861037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:57.861037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:24:57.861042 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:57.861055 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:24:57.861059 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:57.861122 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:57.861211 (+ 89us) server_negotiation.cc:299] Negotiation successful
0621 05:24:57.861256 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 05:24:57.861538 22102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.856739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:46071 (local address 127.2.32.1:43477)
0621 05:24:57.856926 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:57.856960 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:57.857609 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:57.857870 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:57.857874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:57.857883 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:57.858388 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:57.858392 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.859082 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.859084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:57.859860 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:57.859862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.860243 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.860244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:57.860365 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.860886 (+ 521us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:24:57.860886 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:57.860912 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:24:57.860941 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:57.861124 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:57.861130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:57.861133 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:24:57.861213 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 05:24:57.861316 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260621 05:24:57.862092 22100 heartbeater.cc:347] Connected to a master server at 127.2.32.62:46071
I20260621 05:24:57.862166 22100 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:57.862311 22100 heartbeater.cc:511] Master 127.2.32.62:46071 requested a full tablet report, sending...
I20260621 05:24:57.862775 21912 ts_manager.cc:194] Registered new tserver with Master: b637f6e64db64f88b31f3c0a0f29fb4b (127.2.32.1:36387)
I20260621 05:24:57.863400 21912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.32.1:43477
I20260621 05:24:57.863847 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 21972
I20260621 05:24:57.863981 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-0/wal/instance
I20260621 05:24:57.864179 2176 external_mini_cluster.cc:1428] Setting key e2d9d6d7213e8a780daf443aa6d09321
I20260621 05:24:57.865413 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.2:0
--local_ip_for_outbound_sockets=127.2.32.2
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46071
--builtin_ntp_servers=127.2.32.20:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:24:57.945566 22103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:57.945753 22103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:57.945773 22103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:57.946897 22103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:57.946936 22103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:57.946961 22103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.2
I20260621 05:24:57.948275 22103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.32.2
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.2
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:57.948575 22103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:57.948891 22103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:57.953790 22111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:57.953738 22108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:57.953727 22109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:57.953961 22103 server_base.cc:1034] running on GCE node
I20260621 05:24:57.954195 22103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:57.954435 22103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:57.955591 22103 hybrid_clock.cc:648] HybridClock initialized: now 1782019497955587 us; error 40 us; skew 500 ppm
I20260621 05:24:57.956931 22103 webserver.cc:460] Webserver started at http://127.2.32.2:32833/ using document root <none> and password file <none>
I20260621 05:24:57.957168 22103 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:57.957233 22103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:57.957383 22103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:57.958389 22103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-1/data/instance:
uuid: "0e35ac134ab74a5a8cd82a9d005a01f5"
format_stamp: "Formatted at 2026-06-21 05:24:57 on dist-test-slave-b357"
server_key: "08d62bde7ac2b105a1b6b8fc6a7e56b6"
server_key_iv: "903ea0e43ff4095f2cab60d3f0039b39"
server_key_version: "encryptionkey@0"
I20260621 05:24:57.958730 22103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance:
uuid: "0e35ac134ab74a5a8cd82a9d005a01f5"
format_stamp: "Formatted at 2026-06-21 05:24:57 on dist-test-slave-b357"
server_key: "08d62bde7ac2b105a1b6b8fc6a7e56b6"
server_key_iv: "903ea0e43ff4095f2cab60d3f0039b39"
server_key_version: "encryptionkey@0"
I20260621 05:24:57.960121 22103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:24:57.960927 22117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:57.961242 22103 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:57.961347 22103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-1/wal
uuid: "0e35ac134ab74a5a8cd82a9d005a01f5"
format_stamp: "Formatted at 2026-06-21 05:24:57 on dist-test-slave-b357"
server_key: "08d62bde7ac2b105a1b6b8fc6a7e56b6"
server_key_iv: "903ea0e43ff4095f2cab60d3f0039b39"
server_key_version: "encryptionkey@0"
I20260621 05:24:57.961428 22103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:57.972532 22103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:57.972919 22103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:57.973088 22103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:57.973363 22103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:57.973773 22103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:57.973842 22103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:57.973888 22103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:57.973914 22103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:57.981091 22103 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.2:43053
I20260621 05:24:57.981133 22230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.2:43053 every 8 connection(s)
I20260621 05:24:57.981460 22103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-1/data/info.pb
I20260621 05:24:57.987380 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.982183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.2:41863 (local address 127.2.32.62:46071)
0621 05:24:57.982244 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:57.982246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:57.982588 (+ 342us) server_negotiation.cc:407] Connection header received
0621 05:24:57.982978 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:57.982980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:57.982996 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:57.983100 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:57.983894 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.984410 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.985428 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.985660 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.985745 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.986906 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:57.986907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:57.986908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:24:57.986913 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:57.986931 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:24:57.986936 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:57.986999 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:57.987258 (+ 259us) server_negotiation.cc:299] Negotiation successful
0621 05:24:57.987300 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 05:24:57.987543 22233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:57.982141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:46071 (local address 127.2.32.2:41863)
0621 05:24:57.982352 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:57.982399 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:57.982869 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:57.983219 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:57.983224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:57.983234 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:57.983756 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:57.983760 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.984521 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.984522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:57.985301 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:57.985303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:57.985813 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:57.985814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:57.985985 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:57.986718 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:24:57.986719 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:57.986756 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:24:57.986787 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:57.987057 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:57.987063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:57.987066 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:24:57.987238 (+ 172us) client_negotiation.cc:240] Negotiation successful
0621 05:24:57.987336 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":115,"threads_started":1}
I20260621 05:24:57.988157 22231 heartbeater.cc:347] Connected to a master server at 127.2.32.62:46071
I20260621 05:24:57.988240 22231 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:57.988420 22231 heartbeater.cc:511] Master 127.2.32.62:46071 requested a full tablet report, sending...
I20260621 05:24:57.988910 21912 ts_manager.cc:194] Registered new tserver with Master: 0e35ac134ab74a5a8cd82a9d005a01f5 (127.2.32.2:43053)
I20260621 05:24:57.989297 21912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.32.2:41863
I20260621 05:24:57.990980 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 22103
I20260621 05:24:57.991079 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-1/wal/instance
I20260621 05:24:57.991319 2176 external_mini_cluster.cc:1428] Setting key 22fc01f450e89b2f8b9c92d640547c9c
I20260621 05:24:57.992532 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.32.3:0
--local_ip_for_outbound_sockets=127.2.32.3
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46071
--builtin_ntp_servers=127.2.32.20:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 05:24:58.075369 22234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:58.075556 22234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:58.075577 22234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:58.076714 22234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:58.076756 22234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:58.076781 22234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.32.3
I20260621 05:24:58.078161 22234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.32.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.32.3
--webserver_port=0
--tserver_master_addrs=127.2.32.62:46071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.32.3
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:58.078377 22234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:58.078656 22234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:58.083429 22242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:58.083444 22239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:58.083505 22240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:58.083717 22234 server_base.cc:1034] running on GCE node
I20260621 05:24:58.083920 22234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:58.084106 22234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:58.085259 22234 hybrid_clock.cc:648] HybridClock initialized: now 1782019498085243 us; error 31 us; skew 500 ppm
I20260621 05:24:58.086544 22234 webserver.cc:460] Webserver started at http://127.2.32.3:39529/ using document root <none> and password file <none>
I20260621 05:24:58.086795 22234 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:58.086845 22234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:58.086977 22234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:58.088035 22234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-2/data/instance:
uuid: "d79652af6d814f53b407f07542ba18d6"
format_stamp: "Formatted at 2026-06-21 05:24:58 on dist-test-slave-b357"
server_key: "24b4cc5e2e4349ad83384849d4239289"
server_key_iv: "6fdde8ce987f9b026b35f4792ce7f06a"
server_key_version: "encryptionkey@0"
I20260621 05:24:58.088383 22234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance:
uuid: "d79652af6d814f53b407f07542ba18d6"
format_stamp: "Formatted at 2026-06-21 05:24:58 on dist-test-slave-b357"
server_key: "24b4cc5e2e4349ad83384849d4239289"
server_key_iv: "6fdde8ce987f9b026b35f4792ce7f06a"
server_key_version: "encryptionkey@0"
I20260621 05:24:58.089789 22234 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:24:58.090821 22248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:58.091236 22234 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:58.091356 22234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-2/wal
uuid: "d79652af6d814f53b407f07542ba18d6"
format_stamp: "Formatted at 2026-06-21 05:24:58 on dist-test-slave-b357"
server_key: "24b4cc5e2e4349ad83384849d4239289"
server_key_iv: "6fdde8ce987f9b026b35f4792ce7f06a"
server_key_version: "encryptionkey@0"
I20260621 05:24:58.091444 22234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:58.110131 22234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:58.110502 22234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:58.110672 22234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:58.110971 22234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 05:24:58.111452 22234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 05:24:58.111487 22234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:58.111511 22234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 05:24:58.111524 22234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:58.118028 22234 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.3:37715
I20260621 05:24:58.118119 22361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.3:37715 every 8 connection(s)
I20260621 05:24:58.118505 22234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-2/data/info.pb
I20260621 05:24:58.124432 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:58.119335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.3:59323 (local address 127.2.32.62:46071)
0621 05:24:58.119424 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:58.119427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:58.119761 (+ 334us) server_negotiation.cc:407] Connection header received
0621 05:24:58.120340 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:58.120342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:58.120354 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:58.120427 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:58.121214 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.121751 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.122862 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.123089 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.123193 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:58.124115 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:58.124117 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:58.124118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:24:58.124123 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:58.124144 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:24:58.124149 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:58.124213 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:58.124325 (+ 112us) server_negotiation.cc:299] Negotiation successful
0621 05:24:58.124352 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 05:24:58.124563 22364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:58.119310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:46071 (local address 127.2.32.3:59323)
0621 05:24:58.119589 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:58.119630 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:58.120223 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:58.120485 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:58.120489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:58.120498 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:24:58.121110 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:58.121115 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.121889 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.121891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:58.122756 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:58.122759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.123226 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.123227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:58.123351 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:58.123928 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 05:24:58.123928 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:24:58.123962 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 05:24:58.123986 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:24:58.124224 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:24:58.124228 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:24:58.124230 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 05:24:58.124288 (+ 58us) client_negotiation.cc:240] Negotiation successful
0621 05:24:58.124367 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":118,"threads_started":1}
I20260621 05:24:58.125082 22362 heartbeater.cc:347] Connected to a master server at 127.2.32.62:46071
I20260621 05:24:58.125186 22362 heartbeater.cc:464] Registering TS with master...
I20260621 05:24:58.125384 22362 heartbeater.cc:511] Master 127.2.32.62:46071 requested a full tablet report, sending...
I20260621 05:24:58.125790 21912 ts_manager.cc:194] Registered new tserver with Master: d79652af6d814f53b407f07542ba18d6 (127.2.32.3:37715)
I20260621 05:24:58.126227 21912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.32.3:59323
I20260621 05:24:58.128453 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 22234
I20260621 05:24:58.128553 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-2/wal/instance
I20260621 05:24:58.128703 2176 external_mini_cluster.cc:1428] Setting key 0e9ee67404696387a9126263fe09b8a3
I20260621 05:24:58.129642 2176 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 05:24:58.134483 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:58.131278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55646 (local address 127.2.32.62:46071)
0621 05:24:58.131326 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:58.131328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:58.131501 (+ 173us) server_negotiation.cc:407] Connection header received
0621 05:24:58.131560 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:58.131561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:58.131568 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:58.131621 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:58.131895 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.132333 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.133143 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.133317 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.133388 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:58.134174 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:58.134175 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:58.134176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:24:58.134181 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:58.134203 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:24:58.134207 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:58.134272 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:58.134403 (+ 131us) server_negotiation.cc:299] Negotiation successful
0621 05:24:58.134427 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260621 05:24:58.136107 21912 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 05:24:58.138063 21912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 05:24:58.144922 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:58.140710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:43053 (local address 127.0.0.1:37770)
0621 05:24:58.141043 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:58.141083 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:58.141321 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:58.141560 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:58.141560 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:58.141562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:58.141591 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:58.141592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.142516 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.142517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:58.143726 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:58.143728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.144633 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.144634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:58.144715 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:58.144728 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 05:24:58.144797 (+ 69us) client_negotiation.cc:240] Negotiation successful
0621 05:24:58.144849 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":90,"threads_started":1}
I20260621 05:24:58.145067 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:58.140938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:36387 (local address 127.0.0.1:58068)
0621 05:24:58.141201 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:58.141209 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:58.141246 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:58.141538 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:58.141539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:58.141542 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:58.141585 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:58.141587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.142978 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.142979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:58.144021 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:58.144024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.144840 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.144841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:58.144901 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:58.144904 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 05:24:58.144952 (+ 48us) client_negotiation.cc:240] Negotiation successful
0621 05:24:58.144991 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":69,"threads_started":1}
I20260621 05:24:58.144941 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:58.140860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37770 (local address 127.2.32.2:43053)
0621 05:24:58.141044 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:58.141049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:58.141153 (+ 104us) server_negotiation.cc:407] Connection header received
0621 05:24:58.141406 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:58.141410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:58.141427 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:58.141502 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:58.141757 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.142433 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.143812 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.144546 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.144621 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:58.144704 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:58.144821 (+ 117us) server_negotiation.cc:299] Negotiation successful
0621 05:24:58.144858 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":83,"threads_started":1}
I20260621 05:24:58.145306 22375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:58.141159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58068 (local address 127.2.32.1:36387)
0621 05:24:58.141310 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:58.141315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:58.141325 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:24:58.141366 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:58.141368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:58.141384 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:58.141469 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:58.141718 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.142789 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.144113 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.144647 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.144714 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:58.144808 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:58.145161 (+ 353us) server_negotiation.cc:299] Negotiation successful
0621 05:24:58.145211 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260621 05:24:58.145187 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:58.140710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:37715 (local address 127.0.0.1:54090)
0621 05:24:58.141170 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:58.141181 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:58.141246 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:58.141457 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:58.141459 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:58.141461 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:58.141523 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:58.141525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.143215 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.143216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:58.143988 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:58.143990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.144956 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.144957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:58.145027 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:58.145032 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:24:58.145084 (+ 52us) client_negotiation.cc:240] Negotiation successful
0621 05:24:58.145121 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":5,"thread_start_us":89,"threads_started":1}
I20260621 05:24:58.145782 22373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:58.140859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54090 (local address 127.2.32.3:37715)
0621 05:24:58.141120 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:58.141123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:58.141321 (+ 198us) server_negotiation.cc:407] Connection header received
0621 05:24:58.141358 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:58.141361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:58.141386 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:58.141456 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:58.141657 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.142787 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.144087 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.144753 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.144845 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:58.144918 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:58.145642 (+ 724us) server_negotiation.cc:299] Negotiation successful
0621 05:24:58.145682 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":89,"threads_started":1}
I20260621 05:24:58.145865 22165 tablet_service.cc:1467] Processing CreateTablet for tablet b2c975f233d748f49208458ad91d4c36 (DEFAULT_TABLE table=test-table [id=48a3b4bb09844fe68c4c8342c22be2b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:58.146005 22034 tablet_service.cc:1467] Processing CreateTablet for tablet b2c975f233d748f49208458ad91d4c36 (DEFAULT_TABLE table=test-table [id=48a3b4bb09844fe68c4c8342c22be2b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:58.146219 22165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c975f233d748f49208458ad91d4c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:58.146334 22034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c975f233d748f49208458ad91d4c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:58.146373 22296 tablet_service.cc:1467] Processing CreateTablet for tablet b2c975f233d748f49208458ad91d4c36 (DEFAULT_TABLE table=test-table [id=48a3b4bb09844fe68c4c8342c22be2b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 05:24:58.146667 22296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c975f233d748f49208458ad91d4c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:58.149772 22377 tablet_bootstrap.cc:492] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b: Bootstrap starting.
I20260621 05:24:58.149945 22376 tablet_bootstrap.cc:492] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5: Bootstrap starting.
I20260621 05:24:58.150560 22376 tablet_bootstrap.cc:654] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:58.150625 22377 tablet_bootstrap.cc:654] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:58.150874 22376 log.cc:824] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:58.150990 22377 log.cc:824] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:58.151222 22378 tablet_bootstrap.cc:492] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6: Bootstrap starting.
I20260621 05:24:58.151852 22376 tablet_bootstrap.cc:492] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5: No bootstrap required, opened a new log
I20260621 05:24:58.151872 22378 tablet_bootstrap.cc:654] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:58.151940 22376 ts_tablet_manager.cc:1397] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 05:24:58.152033 22377 tablet_bootstrap.cc:492] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b: No bootstrap required, opened a new log
I20260621 05:24:58.152104 22377 ts_tablet_manager.cc:1397] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 05:24:58.152181 22378 log.cc:824] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:58.152832 22378 tablet_bootstrap.cc:492] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6: No bootstrap required, opened a new log
I20260621 05:24:58.152909 22378 ts_tablet_manager.cc:1397] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 05:24:58.154625 22378 raft_consensus.cc:348] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79652af6d814f53b407f07542ba18d6" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37715 } } peers { permanent_uuid: "0e35ac134ab74a5a8cd82a9d005a01f5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 43053 } } peers { permanent_uuid: "b637f6e64db64f88b31f3c0a0f29fb4b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36387 } }
I20260621 05:24:58.154770 22378 raft_consensus.cc:374] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:58.154812 22378 raft_consensus.cc:729] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d79652af6d814f53b407f07542ba18d6, State: Initialized, Role: FOLLOWER
I20260621 05:24:58.154938 22378 consensus_queue.cc:260] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79652af6d814f53b407f07542ba18d6" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37715 } } peers { permanent_uuid: "0e35ac134ab74a5a8cd82a9d005a01f5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 43053 } } peers { permanent_uuid: "b637f6e64db64f88b31f3c0a0f29fb4b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36387 } }
I20260621 05:24:58.154879 22376 raft_consensus.cc:348] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79652af6d814f53b407f07542ba18d6" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37715 } } peers { permanent_uuid: "0e35ac134ab74a5a8cd82a9d005a01f5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 43053 } } peers { permanent_uuid: "b637f6e64db64f88b31f3c0a0f29fb4b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36387 } }
I20260621 05:24:58.154898 22377 raft_consensus.cc:348] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79652af6d814f53b407f07542ba18d6" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37715 } } peers { permanent_uuid: "0e35ac134ab74a5a8cd82a9d005a01f5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 43053 } } peers { permanent_uuid: "b637f6e64db64f88b31f3c0a0f29fb4b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36387 } }
I20260621 05:24:58.155066 22376 raft_consensus.cc:374] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:58.155066 22377 raft_consensus.cc:374] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:58.155103 22376 raft_consensus.cc:729] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e35ac134ab74a5a8cd82a9d005a01f5, State: Initialized, Role: FOLLOWER
I20260621 05:24:58.155103 22377 raft_consensus.cc:729] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b637f6e64db64f88b31f3c0a0f29fb4b, State: Initialized, Role: FOLLOWER
I20260621 05:24:58.155221 22378 ts_tablet_manager.cc:1428] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260621 05:24:58.155292 22377 consensus_queue.cc:260] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79652af6d814f53b407f07542ba18d6" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37715 } } peers { permanent_uuid: "0e35ac134ab74a5a8cd82a9d005a01f5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 43053 } } peers { permanent_uuid: "b637f6e64db64f88b31f3c0a0f29fb4b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36387 } }
I20260621 05:24:58.155292 22376 consensus_queue.cc:260] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79652af6d814f53b407f07542ba18d6" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37715 } } peers { permanent_uuid: "0e35ac134ab74a5a8cd82a9d005a01f5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 43053 } } peers { permanent_uuid: "b637f6e64db64f88b31f3c0a0f29fb4b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36387 } }
I20260621 05:24:58.155334 22362 heartbeater.cc:503] Master 127.2.32.62:46071 was elected leader, sending a full tablet report...
I20260621 05:24:58.155604 22376 ts_tablet_manager.cc:1428] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260621 05:24:58.155781 22100 heartbeater.cc:503] Master 127.2.32.62:46071 was elected leader, sending a full tablet report...
I20260621 05:24:58.156213 22231 heartbeater.cc:503] Master 127.2.32.62:46071 was elected leader, sending a full tablet report...
I20260621 05:24:58.156450 22377 ts_tablet_manager.cc:1428] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260621 05:24:58.203675 22384 raft_consensus.cc:482] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 05:24:58.203874 22384 raft_consensus.cc:504] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79652af6d814f53b407f07542ba18d6" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37715 } } peers { permanent_uuid: "0e35ac134ab74a5a8cd82a9d005a01f5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 43053 } } peers { permanent_uuid: "b637f6e64db64f88b31f3c0a0f29fb4b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36387 } }
I20260621 05:24:58.204407 22384 leader_election.cc:288] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d79652af6d814f53b407f07542ba18d6 (127.2.32.3:37715), 0e35ac134ab74a5a8cd82a9d005a01f5 (127.2.32.2:43053)
I20260621 05:24:58.208920 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:58.204740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:44339 (local address 127.2.32.2:43053)
0621 05:24:58.204791 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:58.204793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:58.204805 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:24:58.204931 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:58.204932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:58.204950 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:58.205035 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:58.205236 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.206055 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.207848 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.208468 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.208560 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:58.208571 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:58.208794 (+ 223us) server_negotiation.cc:299] Negotiation successful
0621 05:24:58.208847 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260621 05:24:58.208920 22373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:58.204779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.32.1:35371 (local address 127.2.32.3:37715)
0621 05:24:58.204850 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:58.204852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:58.205019 (+ 167us) server_negotiation.cc:407] Connection header received
0621 05:24:58.205081 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:58.205082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:58.205098 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:58.205141 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 05:24:58.205292 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.206075 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.207849 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.208468 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.208560 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:58.208571 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:58.208794 (+ 223us) server_negotiation.cc:299] Negotiation successful
0621 05:24:58.208847 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 05:24:58.208912 22102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:58.204672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.2:43053 (local address 127.2.32.1:44339)
0621 05:24:58.204738 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:58.204753 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:58.204807 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:58.205045 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:58.205046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:58.205048 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:58.205144 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:58.205147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.206204 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.206206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:58.207679 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:58.207683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.208581 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.208583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:58.208664 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:58.208669 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:24:58.208743 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 05:24:58.208796 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":48}
I20260621 05:24:58.208917 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:58.204719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.3:37715 (local address 127.2.32.1:35371)
0621 05:24:58.204913 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:58.204922 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:24:58.204981 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:24:58.205156 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:24:58.205156 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:24:58.205158 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 05:24:58.205223 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:58.205225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.206204 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.206206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:58.207680 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:24:58.207683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.208594 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.208595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:24:58.208664 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:58.208669 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 05:24:58.208743 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 05:24:58.208796 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":96,"thread_start_us":124,"threads_started":1}
I20260621 05:24:58.209398 22316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c975f233d748f49208458ad91d4c36" candidate_uuid: "b637f6e64db64f88b31f3c0a0f29fb4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d79652af6d814f53b407f07542ba18d6" is_pre_election: true
I20260621 05:24:58.209420 22185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c975f233d748f49208458ad91d4c36" candidate_uuid: "b637f6e64db64f88b31f3c0a0f29fb4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e35ac134ab74a5a8cd82a9d005a01f5" is_pre_election: true
I20260621 05:24:58.209576 22316 raft_consensus.cc:2454] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b637f6e64db64f88b31f3c0a0f29fb4b in term 0.
I20260621 05:24:58.209583 22185 raft_consensus.cc:2454] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b637f6e64db64f88b31f3c0a0f29fb4b in term 0.
I20260621 05:24:58.209820 21989 leader_election.cc:302] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b637f6e64db64f88b31f3c0a0f29fb4b, d79652af6d814f53b407f07542ba18d6; no voters:
I20260621 05:24:58.209997 22384 raft_consensus.cc:2781] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 05:24:58.210134 22384 raft_consensus.cc:482] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 05:24:58.210175 22384 raft_consensus.cc:3037] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:58.211107 22384 raft_consensus.cc:504] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79652af6d814f53b407f07542ba18d6" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37715 } } peers { permanent_uuid: "0e35ac134ab74a5a8cd82a9d005a01f5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 43053 } } peers { permanent_uuid: "b637f6e64db64f88b31f3c0a0f29fb4b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36387 } }
I20260621 05:24:58.211346 22384 leader_election.cc:288] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b [CANDIDATE]: Term 1 election: Requested vote from peers d79652af6d814f53b407f07542ba18d6 (127.2.32.3:37715), 0e35ac134ab74a5a8cd82a9d005a01f5 (127.2.32.2:43053)
I20260621 05:24:58.211573 22185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c975f233d748f49208458ad91d4c36" candidate_uuid: "b637f6e64db64f88b31f3c0a0f29fb4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e35ac134ab74a5a8cd82a9d005a01f5"
I20260621 05:24:58.211645 22316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c975f233d748f49208458ad91d4c36" candidate_uuid: "b637f6e64db64f88b31f3c0a0f29fb4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d79652af6d814f53b407f07542ba18d6"
I20260621 05:24:58.211647 22185 raft_consensus.cc:3037] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:58.211773 22316 raft_consensus.cc:3037] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:58.212512 22185 raft_consensus.cc:2454] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b637f6e64db64f88b31f3c0a0f29fb4b in term 1.
I20260621 05:24:58.212625 22316 raft_consensus.cc:2454] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b637f6e64db64f88b31f3c0a0f29fb4b in term 1.
I20260621 05:24:58.212695 21987 leader_election.cc:302] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e35ac134ab74a5a8cd82a9d005a01f5, b637f6e64db64f88b31f3c0a0f29fb4b; no voters:
I20260621 05:24:58.212810 22384 raft_consensus.cc:2781] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:24:58.212998 22384 raft_consensus.cc:686] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b [term 1 LEADER]: Becoming Leader. State: Replica: b637f6e64db64f88b31f3c0a0f29fb4b, State: Running, Role: LEADER
I20260621 05:24:58.213130 22384 consensus_queue.cc:237] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79652af6d814f53b407f07542ba18d6" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37715 } } peers { permanent_uuid: "0e35ac134ab74a5a8cd82a9d005a01f5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 43053 } } peers { permanent_uuid: "b637f6e64db64f88b31f3c0a0f29fb4b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36387 } }
I20260621 05:24:58.214272 21911 catalog_manager.cc:5495] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b reported cstate change: term changed from 0 to 1, leader changed from <none> to b637f6e64db64f88b31f3c0a0f29fb4b (127.2.32.1). New cstate: current_term: 1 leader_uuid: "b637f6e64db64f88b31f3c0a0f29fb4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79652af6d814f53b407f07542ba18d6" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e35ac134ab74a5a8cd82a9d005a01f5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 43053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b637f6e64db64f88b31f3c0a0f29fb4b" member_type: VOTER last_known_addr { host: "127.2.32.1" port: 36387 } health_report { overall_health: HEALTHY } } }
W20260621 05:24:58.232066 22232 tablet.cc:2369] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 05:24:58.237583 22375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:58.235117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.2.32.1:36387)
0621 05:24:58.235203 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:58.235205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:58.235220 (+ 15us) server_negotiation.cc:407] Connection header received
0621 05:24:58.235256 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:58.235257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:58.235275 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:58.235343 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:24:58.235601 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.236069 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.236864 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:58.237049 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:58.237106 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:58.237232 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:24:58.237339 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:24:58.237405 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:58.237441 (+ 36us) server_negotiation.cc:299] Negotiation successful
0621 05:24:58.237458 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 05:24:58.239354 22185 raft_consensus.cc:1261] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5 [term 1 FOLLOWER]: Refusing update from remote peer b637f6e64db64f88b31f3c0a0f29fb4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:24:58.239388 22316 raft_consensus.cc:1261] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6 [term 1 FOLLOWER]: Refusing update from remote peer b637f6e64db64f88b31f3c0a0f29fb4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:24:58.239723 22384 consensus_queue.cc:1035] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e35ac134ab74a5a8cd82a9d005a01f5" member_type: VOTER last_known_addr { host: "127.2.32.2" port: 43053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:24:58.239974 22384 consensus_queue.cc:1035] T b2c975f233d748f49208458ad91d4c36 P b637f6e64db64f88b31f3c0a0f29fb4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d79652af6d814f53b407f07542ba18d6" member_type: VOTER last_known_addr { host: "127.2.32.3" port: 37715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:24:58.242841 22391 mvcc.cc:205] Tried to move back new op lower bound from 7299151864785629184 to 7299151864682143744. Current Snapshot: MvccSnapshot[applied={T|T < 7299151864785629184}]
I20260621 05:24:58.244911 21912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:24:58.244992 21912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 05:24:58.245720 21912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49e60f9ef35e45e59bfc524c2257918f: Sending DeleteTablet for 3 replicas of tablet b2c975f233d748f49208458ad91d4c36
I20260621 05:24:58.246034 22296 tablet_service.cc:1514] Processing DeleteTablet for tablet b2c975f233d748f49208458ad91d4c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:24:58 UTC) from {username='slave'} at 127.0.0.1:54090
I20260621 05:24:58.246064 22034 tablet_service.cc:1514] Processing DeleteTablet for tablet b2c975f233d748f49208458ad91d4c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:24:58 UTC) from {username='slave'} at 127.0.0.1:58068
I20260621 05:24:58.246248 22165 tablet_service.cc:1514] Processing DeleteTablet for tablet b2c975f233d748f49208458ad91d4c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 05:24:58 UTC) from {username='slave'} at 127.0.0.1:37770
I20260621 05:24:58.246330 22399 tablet_replica.cc:330] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6: stopping tablet replica
I20260621 05:24:58.246426 22399 raft_consensus.cc:2229] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:24:58.246541 22399 raft_consensus.cc:2258] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:24:58.246631 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 21972
I20260621 05:24:58.247637 22399 ts_tablet_manager.cc:1905] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:24:58.249595 22399 ts_tablet_manager.cc:1918] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:24:58.249673 22399 log.cc:1196] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-2/wal/wals/b2c975f233d748f49208458ad91d4c36
I20260621 05:24:58.249922 22399 ts_tablet_manager.cc:1939] T b2c975f233d748f49208458ad91d4c36 P d79652af6d814f53b407f07542ba18d6: Deleting consensus metadata
I20260621 05:24:58.250018 22401 tablet_replica.cc:330] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5: stopping tablet replica
I20260621 05:24:58.250120 22401 raft_consensus.cc:2229] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 05:24:58.250222 22401 raft_consensus.cc:2258] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 05:24:58.250324 21898 catalog_manager.cc:4841] TS d79652af6d814f53b407f07542ba18d6 (127.2.32.3:37715): tablet b2c975f233d748f49208458ad91d4c36 (table test-table [id=48a3b4bb09844fe68c4c8342c22be2b3]) successfully deleted
I20260621 05:24:58.250614 22401 ts_tablet_manager.cc:1905] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 05:24:58.251886 22401 ts_tablet_manager.cc:1918] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 05:24:58.251942 22401 log.cc:1196] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5: Deleting WAL directory at /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782019225111293-2176-0/minicluster-data/ts-1/wal/wals/b2c975f233d748f49208458ad91d4c36
I20260621 05:24:58.252168 22401 ts_tablet_manager.cc:1939] T b2c975f233d748f49208458ad91d4c36 P 0e35ac134ab74a5a8cd82a9d005a01f5: Deleting consensus metadata
I20260621 05:24:58.252568 21896 catalog_manager.cc:4841] TS 0e35ac134ab74a5a8cd82a9d005a01f5 (127.2.32.2:43053): tablet b2c975f233d748f49208458ad91d4c36 (table test-table [id=48a3b4bb09844fe68c4c8342c22be2b3]) successfully deleted
W20260621 05:24:58.253348 21896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.32.1:36387 (error 108)
I20260621 05:24:58.253612 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 22103
I20260621 05:24:58.254354 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:58.253754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.1:36387 (local address 127.0.0.1:58078)
0621 05:24:58.254272 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0621 05:24:58.254316 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.1:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497}
W20260621 05:24:58.254520 21896 catalog_manager.cc:4584] TS b637f6e64db64f88b31f3c0a0f29fb4b (127.2.32.1:36387): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b2c975f233d748f49208458ad91d4c36: Network error: Client connection negotiation failed: client connection to 127.2.32.1:36387: connect: Connection refused (error 111)
I20260621 05:24:58.258975 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 22234
I20260621 05:24:58.263929 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 21881
2026-06-21T05:24:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (721 ms)
[----------] 6 tests from EncryptionPolicyTest (8326 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:24:58 dist-test-slave-b357 krb5kdc[22405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:24:58 dist-test-slave-b357 krb5kdc[22405](info): set up 2 sockets
Jun 21 05:24:58 dist-test-slave-b357 krb5kdc[22405](info): commencing operation
krb5kdc: starting...
W20260621 05:24:59.355036 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:24:59 dist-test-slave-b357 krb5kdc[22405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:24:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:24:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:24:59.476024 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:24:59.560194 22418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:24:59.560395 22418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:24:59.560417 22418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:24:59.561599 22418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:24:59.561645 22418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:24:59.561658 22418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:24:59.561671 22418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:24:59.561683 22418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:24:59.563068 22418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39831
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:24:59.563273 22418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:24:59.563431 22418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:24:59.568603 22423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:59.568679 22426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:24:59.569013 22424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:24:59.569026 22418 server_base.cc:1034] running on GCE node
I20260621 05:24:59.569319 22418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:24:59.569562 22418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:24:59.570724 22418 hybrid_clock.cc:648] HybridClock initialized: now 1782019499570715 us; error 35 us; skew 500 ppm
Jun 21 05:24:59 dist-test-slave-b357 krb5kdc[22405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:24:59.573791 22418 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:24:59.574291 22418 webserver.cc:460] Webserver started at http://127.0.0.1:40559/ using document root <none> and password file <none>
I20260621 05:24:59.574533 22418 fs_manager.cc:359] Metadata directory not provided
I20260621 05:24:59.574604 22418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:24:59.574754 22418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:24:59.575964 22418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "13949575a4164cb7818a5a7616cc94cd"
format_stamp: "Formatted at 2026-06-21 05:24:59 on dist-test-slave-b357"
server_key: "b795c41d709086ff94930f27e5772198"
server_key_iv: "7e28c73a67b04a13de083b81654b758d"
server_key_version: "encryptionkey@0"
I20260621 05:24:59.576352 22418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "13949575a4164cb7818a5a7616cc94cd"
format_stamp: "Formatted at 2026-06-21 05:24:59 on dist-test-slave-b357"
server_key: "b795c41d709086ff94930f27e5772198"
server_key_iv: "7e28c73a67b04a13de083b81654b758d"
server_key_version: "encryptionkey@0"
I20260621 05:24:59.577708 22418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:24:59.578665 22433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:24:59.579010 22418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:24:59.579130 22418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "13949575a4164cb7818a5a7616cc94cd"
format_stamp: "Formatted at 2026-06-21 05:24:59 on dist-test-slave-b357"
server_key: "b795c41d709086ff94930f27e5772198"
server_key_iv: "7e28c73a67b04a13de083b81654b758d"
server_key_version: "encryptionkey@0"
I20260621 05:24:59.579245 22418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:24:59.601496 22418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:24:59.604295 22418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:24:59.604526 22418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:24:59.609853 22418 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39831
I20260621 05:24:59.610028 22485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39831 every 8 connection(s)
I20260621 05:24:59.610351 22418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:24:59.611402 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 22418
I20260621 05:24:59.611547 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:24:59.611747 22486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:24:59.611799 2176 external_mini_cluster.cc:1428] Setting key 9dbfee375abaacd5beb9250dcf5d0bb2
I20260621 05:24:59.615615 22486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd: Bootstrap starting.
I20260621 05:24:59.616258 22486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd: Neither blocks nor log segments found. Creating new log.
I20260621 05:24:59.616537 22486 log.cc:824] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd: Log is configured to *not* fsync() on all Append() calls
I20260621 05:24:59.617220 22486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd: No bootstrap required, opened a new log
Jun 21 05:24:59 dist-test-slave-b357 krb5kdc[22405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 05:24:59.619446 22486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13949575a4164cb7818a5a7616cc94cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } }
I20260621 05:24:59.619573 22486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:24:59.619596 22486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13949575a4164cb7818a5a7616cc94cd, State: Initialized, Role: FOLLOWER
I20260621 05:24:59.619735 22486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13949575a4164cb7818a5a7616cc94cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } }
I20260621 05:24:59.619809 22486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:24:59.619860 22486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:24:59.619906 22486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:24:59.620617 22486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13949575a4164cb7818a5a7616cc94cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } }
I20260621 05:24:59.620733 22486 leader_election.cc:302] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13949575a4164cb7818a5a7616cc94cd; no voters:
I20260621 05:24:59.620929 22486 leader_election.cc:288] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:24:59.621042 22491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:24:59.621219 22486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:24:59.621393 22491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd [term 1 LEADER]: Becoming Leader. State: Replica: 13949575a4164cb7818a5a7616cc94cd, State: Running, Role: LEADER
I20260621 05:24:59.621502 22491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13949575a4164cb7818a5a7616cc94cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } }
I20260621 05:24:59.621836 22492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13949575a4164cb7818a5a7616cc94cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13949575a4164cb7818a5a7616cc94cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } } }
I20260621 05:24:59.621941 22492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd [sys.catalog]: This master's current role is: LEADER
I20260621 05:24:59.621829 22493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13949575a4164cb7818a5a7616cc94cd. Latest consensus state: current_term: 1 leader_uuid: "13949575a4164cb7818a5a7616cc94cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13949575a4164cb7818a5a7616cc94cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } } }
I20260621 05:24:59.622864 22493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd [sys.catalog]: This master's current role is: LEADER
W20260621 05:24:59.623142 22507 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:24:59.623219 22507 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:24:59.623291 22500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:24:59.623467 22500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:24:59.624464 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:59.612478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43609 (local address 127.0.0.1:39831)
0621 05:24:59.612701 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:59.612712 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:59.612750 (+ 38us) server_negotiation.cc:407] Connection header received
0621 05:24:59.613557 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:59.613568 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:59.613642 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:59.613799 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:59.614755 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:59.615788 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:59.616608 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:59.617023 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:59.617148 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:59.619923 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:59.619927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:59.619927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:59.619939 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:59.622782 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:59.623266 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:59.623269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:59.623269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:59.623378 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:59.623636 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:59.623637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:59.623637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:59.623851 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:59.623956 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:59.624157 (+ 201us) server_negotiation.cc:299] Negotiation successful
0621 05:24:59.624255 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":125,"threads_started":1}
I20260621 05:24:59.625801 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:24:59.627552 22500 catalog_manager.cc:1347] Generated new cluster ID: d1a5b03266fd4507937f6565e5fcb2d4
I20260621 05:24:59.627611 22500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:24:59.635561 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:24:59.627167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53255 (local address 127.0.0.1:39831)
0621 05:24:59.627234 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0621 05:24:59.627236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:24:59.627648 (+ 412us) server_negotiation.cc:407] Connection header received
0621 05:24:59.627733 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:24:59.627735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:24:59.627744 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:24:59.627808 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:24:59.628399 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:59.629431 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:59.630457 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:24:59.630779 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:24:59.630837 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:24:59.632173 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:24:59.632174 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:24:59.632175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:24:59.632181 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:24:59.633862 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:59.634456 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:59.634457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:59.634457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:59.634523 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:24:59.634744 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:24:59.634745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:24:59.634745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:24:59.634923 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:24:59.635031 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 05:24:59.635402 (+ 371us) server_negotiation.cc:299] Negotiation successful
0621 05:24:59.635476 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 05:24:59.637341 22500 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:24:59.638118 22500 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:24:59.644779 22500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13949575a4164cb7818a5a7616cc94cd: Generated new TSK 0
I20260621 05:24:59.644965 22500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:24:59.648942 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 22418
W20260621 05:24:59.653919 22403 connection.cc:535] client connection to 127.0.0.1:39831 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39831): Connection reset by peer (error 104)
2026-06-21T05:24:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:24:59 dist-test-slave-b357 krb5kdc[22517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:24:59 dist-test-slave-b357 krb5kdc[22517](info): set up 2 sockets
Jun 21 05:24:59 dist-test-slave-b357 krb5kdc[22517](info): commencing operation
krb5kdc: starting...
W20260621 05:25:00.707283 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:25:00 dist-test-slave-b357 krb5kdc[22517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:25:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:25:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:25:00.838399 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43461
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:25:00.927930 22530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:00.928169 22530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:00.928208 22530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:00.929322 22530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:00.929387 22530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:00.929414 22530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:00.929437 22530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:00.929466 22530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:00.930878 22530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43461
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:00.931161 22530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:00.931375 22530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:00.936257 22538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:00.936328 22535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:00.936277 22536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:00.936551 22530 server_base.cc:1034] running on GCE node
I20260621 05:25:00.936883 22530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:00.937255 22530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:00.938457 22530 hybrid_clock.cc:648] HybridClock initialized: now 1782019500938400 us; error 95 us; skew 500 ppm
Jun 21 05:25:00 dist-test-slave-b357 krb5kdc[22517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:00.941550 22530 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:25:00.942155 22530 webserver.cc:460] Webserver started at http://127.0.0.1:38153/ using document root <none> and password file <none>
I20260621 05:25:00.942404 22530 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:00.942454 22530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:00.942662 22530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:00.943847 22530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "9024e2b24f854bdb8ff6301d09791173"
format_stamp: "Formatted at 2026-06-21 05:25:00 on dist-test-slave-b357"
server_key: "a1611f70599cf38ff435462c261f268e"
server_key_iv: "ce72fea7c39615c32cf58fea1e24adea"
server_key_version: "encryptionkey@0"
I20260621 05:25:00.944209 22530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "9024e2b24f854bdb8ff6301d09791173"
format_stamp: "Formatted at 2026-06-21 05:25:00 on dist-test-slave-b357"
server_key: "a1611f70599cf38ff435462c261f268e"
server_key_iv: "ce72fea7c39615c32cf58fea1e24adea"
server_key_version: "encryptionkey@0"
I20260621 05:25:00.945636 22530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:25:00.946620 22545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:00.946825 22530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:25:00.946915 22530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "9024e2b24f854bdb8ff6301d09791173"
format_stamp: "Formatted at 2026-06-21 05:25:00 on dist-test-slave-b357"
server_key: "a1611f70599cf38ff435462c261f268e"
server_key_iv: "ce72fea7c39615c32cf58fea1e24adea"
server_key_version: "encryptionkey@0"
I20260621 05:25:00.946974 22530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:00.955965 22530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:00.958176 22530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:00.958288 22530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:00.962201 22530 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43461
I20260621 05:25:00.962260 22597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43461 every 8 connection(s)
I20260621 05:25:00.962568 22530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:25:00.963944 22598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:00.964543 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 22530
I20260621 05:25:00.964713 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:25:00.964946 2176 external_mini_cluster.cc:1428] Setting key 8b4b355a73b6d9a5de1f6c060c350ca4
I20260621 05:25:00.967434 22598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173: Bootstrap starting.
I20260621 05:25:00.968170 22598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:00.968487 22598 log.cc:824] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:00.969352 22598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173: No bootstrap required, opened a new log
I20260621 05:25:00.971067 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:00.965481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51611 (local address 127.0.0.1:43461)
0621 05:25:00.965705 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:00.965713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:00.965737 (+ 24us) server_negotiation.cc:407] Connection header received
0621 05:25:00.966275 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:00.966281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:00.966326 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:00.966524 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:00.966687 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:00.967681 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:00.968497 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:00.969113 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:00.969236 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:00.970182 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:00.970187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:00.970187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:00.970390 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:00.970424 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:00.970436 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:00.970600 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:00.970685 (+ 85us) server_negotiation.cc:299] Negotiation successful
0621 05:25:00.970718 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":99,"threads_started":1}
I20260621 05:25:00.971458 22598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9024e2b24f854bdb8ff6301d09791173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43461 } }
I20260621 05:25:00.971647 22598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:25:00.971690 22598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9024e2b24f854bdb8ff6301d09791173, State: Initialized, Role: FOLLOWER
I20260621 05:25:00.971877 22598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9024e2b24f854bdb8ff6301d09791173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43461 } }
I20260621 05:25:00.971976 22598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:25:00.972026 22598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:25:00.972131 22598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:00.972545 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:25:00.973239 22598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9024e2b24f854bdb8ff6301d09791173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43461 } }
I20260621 05:25:00.973384 22598 leader_election.cc:302] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9024e2b24f854bdb8ff6301d09791173; no voters:
I20260621 05:25:00.973606 22598 leader_election.cc:288] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:25:00.973730 22603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:25:00.973889 22598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:00.973894 22603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173 [term 1 LEADER]: Becoming Leader. State: Replica: 9024e2b24f854bdb8ff6301d09791173, State: Running, Role: LEADER
I20260621 05:25:00.974077 22603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9024e2b24f854bdb8ff6301d09791173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43461 } }
I20260621 05:25:00.974464 22603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9024e2b24f854bdb8ff6301d09791173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9024e2b24f854bdb8ff6301d09791173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43461 } } }
I20260621 05:25:00.974516 22608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9024e2b24f854bdb8ff6301d09791173. Latest consensus state: current_term: 1 leader_uuid: "9024e2b24f854bdb8ff6301d09791173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9024e2b24f854bdb8ff6301d09791173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43461 } } }
I20260621 05:25:00.974596 22603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:00.974709 22608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:00.974987 22617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:25:00.975623 22617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:25:00.977629 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:00.974288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56387 (local address 127.0.0.1:43461)
0621 05:25:00.974333 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:00.974335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:00.974486 (+ 151us) server_negotiation.cc:407] Connection header received
0621 05:25:00.974591 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:00.974592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:00.974602 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:00.974632 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:00.974853 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:00.975595 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:00.976418 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:00.976638 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:00.976726 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:00.977405 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:00.977406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:00.977406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:00.977410 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:00.977425 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:00.977429 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:00.977482 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:00.977542 (+ 60us) server_negotiation.cc:299] Negotiation successful
0621 05:25:00.977551 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260621 05:25:00.980324 22617 catalog_manager.cc:1347] Generated new cluster ID: 91d55e51ae0543fd854ed5bdcefdaee9
I20260621 05:25:00.980404 22617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:25:00.986857 22617 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:25:00.987413 22617 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:25:00.995797 22617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9024e2b24f854bdb8ff6301d09791173: Generated new TSK 0
I20260621 05:25:00.995994 22617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:25:00.999835 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 22530
W20260621 05:25:01.003862 22515 connection.cc:535] client connection to 127.0.0.1:43461 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43461): Connection reset by peer (error 104)
2026-06-21T05:25:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1350 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:25:01 dist-test-slave-b357 krb5kdc[22629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:25:01 dist-test-slave-b357 krb5kdc[22629](info): set up 2 sockets
Jun 21 05:25:01 dist-test-slave-b357 krb5kdc[22629](info): commencing operation
krb5kdc: starting...
W20260621 05:25:02.059233 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:25:02 dist-test-slave-b357 krb5kdc[22629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:25:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:25:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:25:02.191287 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37533
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:25:02.275719 22642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:02.275930 22642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:02.275954 22642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:02.277079 22642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:02.277123 22642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:02.277138 22642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:02.277150 22642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:02.277163 22642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:02.278604 22642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37533
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:02.278838 22642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:02.279095 22642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:02.284004 22650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:02.284024 22647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:02.284054 22648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:02.284314 22642 server_base.cc:1034] running on GCE node
I20260621 05:25:02.284552 22642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:02.284833 22642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:02.286015 22642 hybrid_clock.cc:648] HybridClock initialized: now 1782019502286036 us; error 110 us; skew 500 ppm
Jun 21 05:25:02 dist-test-slave-b357 krb5kdc[22629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:02.288686 22642 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:25:02.289178 22642 webserver.cc:460] Webserver started at http://127.0.0.1:39019/ using document root <none> and password file <none>
I20260621 05:25:02.289417 22642 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:02.289485 22642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:02.289636 22642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:02.291102 22642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "66dc1672dee3419d888c5cdf0b1891fc"
format_stamp: "Formatted at 2026-06-21 05:25:02 on dist-test-slave-b357"
server_key: "c42ce0975cf535bf302a82751e4eae0c"
server_key_iv: "b0ba8bbf232ba092781b217fe81b510a"
server_key_version: "encryptionkey@0"
I20260621 05:25:02.291700 22642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "66dc1672dee3419d888c5cdf0b1891fc"
format_stamp: "Formatted at 2026-06-21 05:25:02 on dist-test-slave-b357"
server_key: "c42ce0975cf535bf302a82751e4eae0c"
server_key_iv: "b0ba8bbf232ba092781b217fe81b510a"
server_key_version: "encryptionkey@0"
I20260621 05:25:02.293557 22642 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260621 05:25:02.294457 22657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:02.294752 22642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:25:02.294858 22642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "66dc1672dee3419d888c5cdf0b1891fc"
format_stamp: "Formatted at 2026-06-21 05:25:02 on dist-test-slave-b357"
server_key: "c42ce0975cf535bf302a82751e4eae0c"
server_key_iv: "b0ba8bbf232ba092781b217fe81b510a"
server_key_version: "encryptionkey@0"
I20260621 05:25:02.294953 22642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:02.298370 22642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:02.301067 22642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:02.301311 22642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:02.305889 22642 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37533
I20260621 05:25:02.305917 22709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37533 every 8 connection(s)
I20260621 05:25:02.306275 22642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:25:02.307098 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 22642
I20260621 05:25:02.307281 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:25:02.307477 2176 external_mini_cluster.cc:1428] Setting key ee06cabd76df1f951a00a85f34648426
I20260621 05:25:02.307652 22710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:02.309990 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:02.308018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55479 (local address 127.0.0.1:37533)
0621 05:25:02.308219 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:02.308230 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:02.308260 (+ 30us) server_negotiation.cc:407] Connection header received
0621 05:25:02.308789 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:02.308795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:02.308819 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:02.308931 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:02.309556 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:02.309558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:02.309559 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:02.309562 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:02.309594 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:02.309604 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:02.309657 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:02.309745 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 05:25:02.309799 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":103,"threads_started":1}
I20260621 05:25:02.310992 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:25:02.311115 22710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc: Bootstrap starting.
I20260621 05:25:02.311826 22710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:02.312163 22710 log.cc:824] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:02.313154 22710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc: No bootstrap required, opened a new log
I20260621 05:25:02.314134 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:02.312752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39471 (local address 127.0.0.1:37533)
0621 05:25:02.312786 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:02.312788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:02.313045 (+ 257us) server_negotiation.cc:407] Connection header received
0621 05:25:02.313127 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:02.313128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:02.313137 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:02.313213 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:02.313791 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:02.313793 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:02.313793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:02.313797 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:02.313813 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:02.313817 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:02.313867 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:02.313968 (+ 101us) server_negotiation.cc:299] Negotiation successful
0621 05:25:02.313998 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:25:02.315407 22710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66dc1672dee3419d888c5cdf0b1891fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37533 } }
I20260621 05:25:02.315538 22710 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:25:02.315560 22710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66dc1672dee3419d888c5cdf0b1891fc, State: Initialized, Role: FOLLOWER
I20260621 05:25:02.315672 22710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66dc1672dee3419d888c5cdf0b1891fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37533 } }
I20260621 05:25:02.315757 22710 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:25:02.315779 22710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:25:02.315806 22710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:02.316282 22710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66dc1672dee3419d888c5cdf0b1891fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37533 } }
I20260621 05:25:02.316483 22710 leader_election.cc:302] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66dc1672dee3419d888c5cdf0b1891fc; no voters:
I20260621 05:25:02.316742 22710 leader_election.cc:288] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:25:02.316862 22720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:25:02.317092 22720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc [term 1 LEADER]: Becoming Leader. State: Replica: 66dc1672dee3419d888c5cdf0b1891fc, State: Running, Role: LEADER
I20260621 05:25:02.317158 22710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:02.317250 22720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66dc1672dee3419d888c5cdf0b1891fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37533 } }
I20260621 05:25:02.317737 22721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66dc1672dee3419d888c5cdf0b1891fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66dc1672dee3419d888c5cdf0b1891fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37533 } } }
I20260621 05:25:02.318176 22721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:02.317786 22722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66dc1672dee3419d888c5cdf0b1891fc. Latest consensus state: current_term: 1 leader_uuid: "66dc1672dee3419d888c5cdf0b1891fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66dc1672dee3419d888c5cdf0b1891fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37533 } } }
I20260621 05:25:02.318364 22722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:02.318593 22730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:25:02.319134 22730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:25:02.321030 22730 catalog_manager.cc:1347] Generated new cluster ID: e0fecf615eda44b4b825fed9051364c0
I20260621 05:25:02.321085 22730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:25:02.329527 22730 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:25:02.329685 22730 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:25:02.333438 22730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66dc1672dee3419d888c5cdf0b1891fc: Generated new TSK 0
I20260621 05:25:02.333612 22730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:25:02.338642 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 22642
2026-06-21T05:25:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:25:02 dist-test-slave-b357 krb5kdc[22741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:25:02 dist-test-slave-b357 krb5kdc[22741](info): set up 2 sockets
Jun 21 05:25:02 dist-test-slave-b357 krb5kdc[22741](info): commencing operation
krb5kdc: starting...
W20260621 05:25:03.411562 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:25:03 dist-test-slave-b357 krb5kdc[22741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:25:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:25:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:25:03.542598 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38123
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:25:03.639703 22754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:03.639892 22754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:03.639914 22754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:03.641006 22754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:03.641047 22754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:03.641062 22754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:03.641073 22754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:03.641085 22754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:03.642491 22754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:03.642701 22754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:03.642918 22754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:03.647683 22759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:03.647661 22760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:03.647661 22762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:03.647995 22754 server_base.cc:1034] running on GCE node
I20260621 05:25:03.648240 22754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:03.648514 22754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:03.649727 22754 hybrid_clock.cc:648] HybridClock initialized: now 1782019503649710 us; error 27 us; skew 500 ppm
Jun 21 05:25:03 dist-test-slave-b357 krb5kdc[22741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:03.652498 22754 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:25:03.652906 22754 webserver.cc:460] Webserver started at http://127.0.0.1:45825/ using document root <none> and password file <none>
I20260621 05:25:03.653137 22754 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:03.653206 22754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:03.653357 22754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:03.654387 22754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "454dbc462d1b436b819562748e6f3b61"
format_stamp: "Formatted at 2026-06-21 05:25:03 on dist-test-slave-b357"
server_key: "81b50fd26d1e16c8f6c5af7df7ec9965"
server_key_iv: "259fbe861e94bf9d7dd52543f4b9ea8c"
server_key_version: "encryptionkey@0"
I20260621 05:25:03.654737 22754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "454dbc462d1b436b819562748e6f3b61"
format_stamp: "Formatted at 2026-06-21 05:25:03 on dist-test-slave-b357"
server_key: "81b50fd26d1e16c8f6c5af7df7ec9965"
server_key_iv: "259fbe861e94bf9d7dd52543f4b9ea8c"
server_key_version: "encryptionkey@0"
I20260621 05:25:03.656286 22754 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:25:03.657410 22769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:03.657766 22754 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:25:03.657896 22754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "454dbc462d1b436b819562748e6f3b61"
format_stamp: "Formatted at 2026-06-21 05:25:03 on dist-test-slave-b357"
server_key: "81b50fd26d1e16c8f6c5af7df7ec9965"
server_key_iv: "259fbe861e94bf9d7dd52543f4b9ea8c"
server_key_version: "encryptionkey@0"
I20260621 05:25:03.657982 22754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:03.683199 22754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:03.685701 22754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:03.685870 22754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:03.689925 22754 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38123
I20260621 05:25:03.689949 22821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38123 every 8 connection(s)
I20260621 05:25:03.690290 22754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:25:03.691265 22822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:03.693723 22822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61: Bootstrap starting.
I20260621 05:25:03.694299 22822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:03.694593 22822 log.cc:824] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:03.695866 22822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61: No bootstrap required, opened a new log
I20260621 05:25:03.697860 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 22754
I20260621 05:25:03.697777 22822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454dbc462d1b436b819562748e6f3b61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38123 } }
I20260621 05:25:03.697986 22822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:25:03.698009 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:25:03.698040 22822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 454dbc462d1b436b819562748e6f3b61, State: Initialized, Role: FOLLOWER
I20260621 05:25:03.698210 2176 external_mini_cluster.cc:1428] Setting key ab9f25f847343ce2dcef8557ddc6b34f
I20260621 05:25:03.698220 22822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454dbc462d1b436b819562748e6f3b61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38123 } }
I20260621 05:25:03.698307 22822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:25:03.698352 22822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:25:03.698408 22822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:03.699265 22822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454dbc462d1b436b819562748e6f3b61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38123 } }
I20260621 05:25:03.699407 22822 leader_election.cc:302] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 454dbc462d1b436b819562748e6f3b61; no voters:
I20260621 05:25:03.699584 22822 leader_election.cc:288] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:25:03.699630 22827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:25:03.699761 22827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61 [term 1 LEADER]: Becoming Leader. State: Replica: 454dbc462d1b436b819562748e6f3b61, State: Running, Role: LEADER
I20260621 05:25:03.699898 22822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:03.699939 22827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454dbc462d1b436b819562748e6f3b61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38123 } }
I20260621 05:25:03.700680 22828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 454dbc462d1b436b819562748e6f3b61. Latest consensus state: current_term: 1 leader_uuid: "454dbc462d1b436b819562748e6f3b61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454dbc462d1b436b819562748e6f3b61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38123 } } }
I20260621 05:25:03.700834 22828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:03.701081 22829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "454dbc462d1b436b819562748e6f3b61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454dbc462d1b436b819562748e6f3b61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38123 } } }
I20260621 05:25:03.701269 22829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:03.701479 22836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:25:03.702118 22836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:25:03.704241 22836 catalog_manager.cc:1347] Generated new cluster ID: 2eea1940fba54f2db4ed76260b55fdc6
I20260621 05:25:03.704362 22836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:25:03 dist-test-slave-b357 krb5kdc[22741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 05:25:03.711834 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:03.698779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37883 (local address 127.0.0.1:38123)
0621 05:25:03.699534 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:03.699555 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:03.699604 (+ 49us) server_negotiation.cc:407] Connection header received
0621 05:25:03.699822 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:03.699833 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:03.699905 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:03.700251 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:03.701262 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:03.702575 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:03.703625 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:03.704233 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:03.704352 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:03.707385 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:03.707390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:03.707391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:03.707406 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:03.709872 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:03.710297 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:03.710300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:03.710300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:03.710416 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:03.710654 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:03.710655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:03.710655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:03.710936 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:03.711124 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:03.711526 (+ 402us) server_negotiation.cc:299] Negotiation successful
0621 05:25:03.711664 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":537,"threads_started":1}
I20260621 05:25:03.712849 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:25:03.721566 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:03.714348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49859 (local address 127.0.0.1:38123)
0621 05:25:03.714389 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:03.714391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:03.714524 (+ 133us) server_negotiation.cc:407] Connection header received
0621 05:25:03.714605 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:03.714606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:03.714619 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:03.714671 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:03.715377 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:03.715923 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:03.716704 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:03.717177 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:03.717315 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:03.718608 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:03.718609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:03.718609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:03.718616 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:03.720072 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:03.720429 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:03.720430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:03.720430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:03.720481 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:03.720876 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:03.720879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:03.720879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:03.721113 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:03.721262 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:03.721380 (+ 118us) server_negotiation.cc:299] Negotiation successful
0621 05:25:03.721454 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:25:03.723640 22836 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:25:03.724311 22836 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:25:03.733321 22836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 454dbc462d1b436b819562748e6f3b61: Generated new TSK 0
I20260621 05:25:03.733644 22836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:25:03.740026 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 22754
2026-06-21T05:25:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:25:03 dist-test-slave-b357 krb5kdc[22853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:25:03 dist-test-slave-b357 krb5kdc[22853](info): set up 2 sockets
Jun 21 05:25:03 dist-test-slave-b357 krb5kdc[22853](info): commencing operation
krb5kdc: starting...
W20260621 05:25:04.803471 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:25:04 dist-test-slave-b357 krb5kdc[22853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:25:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:25:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:25:04.934065 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:25:05.018241 22866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:05.018452 22866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:05.018476 22866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:05.019702 22866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:05.019757 22866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:05.019773 22866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:05.019785 22866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:05.019804 22866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:05.021224 22866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:05.021461 22866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:05.021695 22866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:05.026664 22871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:05.026808 22874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:05.026664 22872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:05.027172 22866 server_base.cc:1034] running on GCE node
I20260621 05:25:05.027513 22866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:05.027889 22866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:05.029075 22866 hybrid_clock.cc:648] HybridClock initialized: now 1782019505029044 us; error 81 us; skew 500 ppm
Jun 21 05:25:05 dist-test-slave-b357 krb5kdc[22853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:05.032308 22866 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:25:05.032886 22866 webserver.cc:460] Webserver started at http://127.0.0.1:34893/ using document root <none> and password file <none>
I20260621 05:25:05.033157 22866 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:05.033228 22866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:05.033392 22866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:05.034575 22866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "62c558f6d45c4251803cecd807b66e0e"
format_stamp: "Formatted at 2026-06-21 05:25:05 on dist-test-slave-b357"
server_key: "b0e5fd84883454ad02984f7157962b62"
server_key_iv: "b15469a8b0cf52a36a479dbf7922d801"
server_key_version: "encryptionkey@0"
I20260621 05:25:05.034953 22866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "62c558f6d45c4251803cecd807b66e0e"
format_stamp: "Formatted at 2026-06-21 05:25:05 on dist-test-slave-b357"
server_key: "b0e5fd84883454ad02984f7157962b62"
server_key_iv: "b15469a8b0cf52a36a479dbf7922d801"
server_key_version: "encryptionkey@0"
I20260621 05:25:05.036500 22866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:25:05.037426 22881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:05.037732 22866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 05:25:05.037845 22866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "62c558f6d45c4251803cecd807b66e0e"
format_stamp: "Formatted at 2026-06-21 05:25:05 on dist-test-slave-b357"
server_key: "b0e5fd84883454ad02984f7157962b62"
server_key_iv: "b15469a8b0cf52a36a479dbf7922d801"
server_key_version: "encryptionkey@0"
I20260621 05:25:05.037931 22866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:05.056365 22866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:05.058858 22866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:05.059031 22866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:05.063366 22866 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37041
I20260621 05:25:05.063406 22933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37041 every 8 connection(s)
I20260621 05:25:05.063804 22866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:25:05.064740 22934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:05.067003 22934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e: Bootstrap starting.
I20260621 05:25:05.067603 22934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:05.067902 22934 log.cc:824] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:05.068595 22934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e: No bootstrap required, opened a new log
I20260621 05:25:05.070191 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 22866
I20260621 05:25:05.070319 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:25:05.070302 22934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c558f6d45c4251803cecd807b66e0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37041 } }
I20260621 05:25:05.070492 2176 external_mini_cluster.cc:1428] Setting key 9acfd7aea21e7e8728b2655b7dbc0148
I20260621 05:25:05.070482 22934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:25:05.070523 22934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c558f6d45c4251803cecd807b66e0e, State: Initialized, Role: FOLLOWER
I20260621 05:25:05.070657 22934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c558f6d45c4251803cecd807b66e0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37041 } }
I20260621 05:25:05.070745 22934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:25:05.070768 22934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:25:05.070840 22934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:05.071934 22934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c558f6d45c4251803cecd807b66e0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37041 } }
I20260621 05:25:05.072019 22934 leader_election.cc:302] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62c558f6d45c4251803cecd807b66e0e; no voters:
I20260621 05:25:05.072234 22934 leader_election.cc:288] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:25:05.072263 22939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:25:05.072366 22939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e [term 1 LEADER]: Becoming Leader. State: Replica: 62c558f6d45c4251803cecd807b66e0e, State: Running, Role: LEADER
I20260621 05:25:05.072487 22934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:05.072497 22939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c558f6d45c4251803cecd807b66e0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37041 } }
I20260621 05:25:05.073086 22941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62c558f6d45c4251803cecd807b66e0e. Latest consensus state: current_term: 1 leader_uuid: "62c558f6d45c4251803cecd807b66e0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c558f6d45c4251803cecd807b66e0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37041 } } }
I20260621 05:25:05.073228 22941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:05.073062 22940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62c558f6d45c4251803cecd807b66e0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c558f6d45c4251803cecd807b66e0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37041 } } }
I20260621 05:25:05.073617 22940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:05.073659 22944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:25:05.074594 22944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:25:05.077220 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:05.070997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49563 (local address 127.0.0.1:37041)
0621 05:25:05.071237 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:05.071246 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:05.071282 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:25:05.071386 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:05.071391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:05.071438 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:05.071557 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:05.071838 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:05.073101 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:05.074540 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:05.075427 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:05.075549 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:05.076322 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:05.076325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:05.076326 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:05.076548 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:05.076585 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:05.076599 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:05.076661 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:05.076947 (+ 286us) server_negotiation.cc:299] Negotiation successful
0621 05:25:05.077013 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":127,"threads_started":1}
I20260621 05:25:05.077626 22944 catalog_manager.cc:1347] Generated new cluster ID: 41e56f2758a940669bf7f4517a74d127
I20260621 05:25:05.077734 22944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:25:05.078122 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:25:05.086547 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:05.083697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39333 (local address 127.0.0.1:37041)
0621 05:25:05.083746 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:05.083748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:05.083882 (+ 134us) server_negotiation.cc:407] Connection header received
0621 05:25:05.083956 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:05.083957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:05.083968 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:05.084018 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:05.084149 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:05.084625 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:05.085315 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:05.085529 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:05.085605 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:05.086270 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:05.086271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:05.086272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:05.086276 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:05.086289 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:05.086293 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:05.086346 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:05.086415 (+ 69us) server_negotiation.cc:299] Negotiation successful
0621 05:25:05.086439 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":20}
I20260621 05:25:05.100274 22944 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:25:05.100979 22944 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:25:05.105021 22944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62c558f6d45c4251803cecd807b66e0e: Generated new TSK 0
I20260621 05:25:05.105187 22944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:25:05.109341 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 22866
2026-06-21T05:25:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:25:05 dist-test-slave-b357 krb5kdc[22965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:25:05 dist-test-slave-b357 krb5kdc[22965](info): set up 2 sockets
Jun 21 05:25:05 dist-test-slave-b357 krb5kdc[22965](info): commencing operation
krb5kdc: starting...
W20260621 05:25:06.159827 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:25:06 dist-test-slave-b357 krb5kdc[22965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:25:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:25:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:25:06.299733 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39873
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:25:06.380708 22978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:06.380944 22978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:06.380980 22978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:06.382090 22978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:06.382156 22978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:06.382182 22978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:06.382215 22978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:06.382248 22978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:06.383821 22978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:06.384081 22978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:06.384310 22978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:06.390025 22984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:06.390027 22986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:06.390025 22983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:06.390260 22978 server_base.cc:1034] running on GCE node
I20260621 05:25:06.390594 22978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:06.390973 22978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:06.392148 22978 hybrid_clock.cc:648] HybridClock initialized: now 1782019506392107 us; error 55 us; skew 500 ppm
Jun 21 05:25:06 dist-test-slave-b357 krb5kdc[22965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:06.399773 22978 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:25:06.400291 22978 webserver.cc:460] Webserver started at http://127.0.0.1:34151/ using document root <none> and password file <none>
I20260621 05:25:06.400516 22978 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:06.400581 22978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:06.400731 22978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:06.401791 22978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "604ba17b8c494939b4df9b6244899491"
format_stamp: "Formatted at 2026-06-21 05:25:06 on dist-test-slave-b357"
server_key: "d2244269008cdec6bc77e25c9dc9e041"
server_key_iv: "589e82aa9905ff033af8e516f41d4759"
server_key_version: "encryptionkey@0"
I20260621 05:25:06.402148 22978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "604ba17b8c494939b4df9b6244899491"
format_stamp: "Formatted at 2026-06-21 05:25:06 on dist-test-slave-b357"
server_key: "d2244269008cdec6bc77e25c9dc9e041"
server_key_iv: "589e82aa9905ff033af8e516f41d4759"
server_key_version: "encryptionkey@0"
I20260621 05:25:06.403789 22978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:25:06.405124 22993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:06.405408 22978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:25:06.405512 22978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "604ba17b8c494939b4df9b6244899491"
format_stamp: "Formatted at 2026-06-21 05:25:06 on dist-test-slave-b357"
server_key: "d2244269008cdec6bc77e25c9dc9e041"
server_key_iv: "589e82aa9905ff033af8e516f41d4759"
server_key_version: "encryptionkey@0"
I20260621 05:25:06.405567 22978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:06.408787 22978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:06.410987 22978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:06.411193 22978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:06.415117 22978 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39873
I20260621 05:25:06.415194 23045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39873 every 8 connection(s)
I20260621 05:25:06.415529 22978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:25:06.416536 23046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:06.418740 23046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491: Bootstrap starting.
I20260621 05:25:06.419378 23046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:06.419668 23046 log.cc:824] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:06.420398 23046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491: No bootstrap required, opened a new log
I20260621 05:25:06.421983 23046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "604ba17b8c494939b4df9b6244899491" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39873 } }
I20260621 05:25:06.422114 23046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:25:06.422137 23046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604ba17b8c494939b4df9b6244899491, State: Initialized, Role: FOLLOWER
I20260621 05:25:06.422274 23046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "604ba17b8c494939b4df9b6244899491" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39873 } }
I20260621 05:25:06.422358 23046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:25:06.422380 23046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:25:06.422410 23046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:06.422988 23046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "604ba17b8c494939b4df9b6244899491" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39873 } }
I20260621 05:25:06.423079 23046 leader_election.cc:302] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 604ba17b8c494939b4df9b6244899491; no voters:
I20260621 05:25:06.423282 23046 leader_election.cc:288] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:25:06.423571 23049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:25:06.423591 23046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:06.423933 23049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491 [term 1 LEADER]: Becoming Leader. State: Replica: 604ba17b8c494939b4df9b6244899491, State: Running, Role: LEADER
I20260621 05:25:06.424132 23049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "604ba17b8c494939b4df9b6244899491" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39873 } }
I20260621 05:25:06.424741 23051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 604ba17b8c494939b4df9b6244899491. Latest consensus state: current_term: 1 leader_uuid: "604ba17b8c494939b4df9b6244899491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "604ba17b8c494939b4df9b6244899491" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39873 } } }
I20260621 05:25:06.424850 23051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:06.424870 23050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "604ba17b8c494939b4df9b6244899491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "604ba17b8c494939b4df9b6244899491" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39873 } } }
I20260621 05:25:06.424922 23050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:06.425120 23061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:25:06.425396 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 22978
I20260621 05:25:06.425515 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:25:06.425657 2176 external_mini_cluster.cc:1428] Setting key f80e68432aa6f4ec965dc876b7e3ca6b
I20260621 05:25:06.425765 23061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:25:06.427206 23061 catalog_manager.cc:1347] Generated new cluster ID: 621cec2e4d334cffafc0c8b93be92d56
I20260621 05:25:06.427263 23061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:25:06.428093 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:06.426104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34715 (local address 127.0.0.1:39873)
0621 05:25:06.426270 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:06.426279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:06.426598 (+ 319us) server_negotiation.cc:407] Connection header received
0621 05:25:06.426739 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:06.426746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:06.426780 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:06.426932 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:06.427689 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:06.427692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:06.427692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:06.427696 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:06.427736 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:06.427750 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:06.427811 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:06.427902 (+ 91us) server_negotiation.cc:299] Negotiation successful
0621 05:25:06.427938 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260621 05:25:06.432765 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:25:06.435662 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:06.434625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56891 (local address 127.0.0.1:39873)
0621 05:25:06.434657 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:06.434659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:06.434775 (+ 116us) server_negotiation.cc:407] Connection header received
0621 05:25:06.434840 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:06.434841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:06.434845 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:06.434905 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:06.435460 (+ 555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:06.435461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:06.435461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:06.435465 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:06.435478 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:06.435482 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:06.435533 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:06.435588 (+ 55us) server_negotiation.cc:299] Negotiation successful
0621 05:25:06.435601 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260621 05:25:06.440654 23061 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:25:06.440814 23061 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:25:06.454391 23061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 604ba17b8c494939b4df9b6244899491: Generated new TSK 0
I20260621 05:25:06.454730 23061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:25:06.458514 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 22978
2026-06-21T05:25:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1350 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:25:06 dist-test-slave-b357 krb5kdc[23077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:25:06 dist-test-slave-b357 krb5kdc[23077](info): set up 2 sockets
Jun 21 05:25:06 dist-test-slave-b357 krb5kdc[23077](info): commencing operation
krb5kdc: starting...
W20260621 05:25:07.524555 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:25:07 dist-test-slave-b357 krb5kdc[23077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:25:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:25:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:25:07.651320 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44759
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:25:07.739777 23090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:07.740078 23090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:07.740129 23090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:07.741447 23090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:07.741518 23090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:07.741546 23090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:07.741577 23090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:07.741603 23090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:07.743151 23090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:07.743441 23090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:07.743687 23090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:07.748667 23096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:07.748675 23095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:07.748675 23098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:07.748917 23090 server_base.cc:1034] running on GCE node
I20260621 05:25:07.749145 23090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:07.749375 23090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:07.750571 23090 hybrid_clock.cc:648] HybridClock initialized: now 1782019507750542 us; error 31 us; skew 500 ppm
Jun 21 05:25:07 dist-test-slave-b357 krb5kdc[23077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:07.753870 23090 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:25:07.754329 23090 webserver.cc:460] Webserver started at http://127.0.0.1:39553/ using document root <none> and password file <none>
I20260621 05:25:07.754567 23090 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:07.754642 23090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:07.754807 23090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:07.755985 23090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "c97f6ae0cf484807aec50b8ff30e9852"
format_stamp: "Formatted at 2026-06-21 05:25:07 on dist-test-slave-b357"
server_key: "d0ec27d8b4060b4133190477e305de00"
server_key_iv: "4f57ed3b390ec51a159e6b7729ce7837"
server_key_version: "encryptionkey@0"
I20260621 05:25:07.756430 23090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "c97f6ae0cf484807aec50b8ff30e9852"
format_stamp: "Formatted at 2026-06-21 05:25:07 on dist-test-slave-b357"
server_key: "d0ec27d8b4060b4133190477e305de00"
server_key_iv: "4f57ed3b390ec51a159e6b7729ce7837"
server_key_version: "encryptionkey@0"
I20260621 05:25:07.757958 23090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:25:07.758893 23105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:07.759199 23090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:25:07.759307 23090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "c97f6ae0cf484807aec50b8ff30e9852"
format_stamp: "Formatted at 2026-06-21 05:25:07 on dist-test-slave-b357"
server_key: "d0ec27d8b4060b4133190477e305de00"
server_key_iv: "4f57ed3b390ec51a159e6b7729ce7837"
server_key_version: "encryptionkey@0"
I20260621 05:25:07.759397 23090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:07.785176 23090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:07.787593 23090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:07.787792 23090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:07.792265 23090 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44759
I20260621 05:25:07.792315 23157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44759 every 8 connection(s)
I20260621 05:25:07.792699 23090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:25:07.793648 23158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:07.795955 23158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852: Bootstrap starting.
I20260621 05:25:07.796545 23158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:07.796851 23158 log.cc:824] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:07.796923 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 23090
I20260621 05:25:07.797210 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:25:07.797463 2176 external_mini_cluster.cc:1428] Setting key fac60df29e2c216b19332e5dc92ff42a
I20260621 05:25:07.797576 23158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852: No bootstrap required, opened a new log
I20260621 05:25:07.799767 23158 raft_consensus.cc:348] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97f6ae0cf484807aec50b8ff30e9852" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44759 } }
I20260621 05:25:07.799934 23158 raft_consensus.cc:374] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:25:07.799971 23158 raft_consensus.cc:729] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97f6ae0cf484807aec50b8ff30e9852, State: Initialized, Role: FOLLOWER
I20260621 05:25:07.800098 23158 consensus_queue.cc:260] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97f6ae0cf484807aec50b8ff30e9852" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44759 } }
I20260621 05:25:07.800189 23158 raft_consensus.cc:388] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:25:07.800239 23158 raft_consensus.cc:482] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:25:07.800287 23158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:07.800925 23158 raft_consensus.cc:504] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97f6ae0cf484807aec50b8ff30e9852" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44759 } }
I20260621 05:25:07.801025 23158 leader_election.cc:302] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c97f6ae0cf484807aec50b8ff30e9852; no voters:
I20260621 05:25:07.801156 23158 leader_election.cc:288] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:25:07.801236 23163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:25:07.801378 23158 sys_catalog.cc:564] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:07.801429 23163 raft_consensus.cc:686] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852 [term 1 LEADER]: Becoming Leader. State: Replica: c97f6ae0cf484807aec50b8ff30e9852, State: Running, Role: LEADER
I20260621 05:25:07.801542 23163 consensus_queue.cc:237] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97f6ae0cf484807aec50b8ff30e9852" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44759 } }
W20260621 05:25:07.803186 23178 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:25:07.803242 23178 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:25:07.803395 23165 sys_catalog.cc:455] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c97f6ae0cf484807aec50b8ff30e9852. Latest consensus state: current_term: 1 leader_uuid: "c97f6ae0cf484807aec50b8ff30e9852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97f6ae0cf484807aec50b8ff30e9852" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44759 } } }
I20260621 05:25:07.803483 23165 sys_catalog.cc:458] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:07.803666 23180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:25:07.803882 23180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:25:07.803889 23164 sys_catalog.cc:455] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c97f6ae0cf484807aec50b8ff30e9852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97f6ae0cf484807aec50b8ff30e9852" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44759 } } }
I20260621 05:25:07.804008 23164 sys_catalog.cc:458] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:07.805334 23180 catalog_manager.cc:1347] Generated new cluster ID: a9891831a68a4b579caf9be7391554db
I20260621 05:25:07.805382 23180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:25:07 dist-test-slave-b357 krb5kdc[23077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 05:25:07.811076 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:07.798072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:60685 (local address 127.0.0.1:44759)
0621 05:25:07.798289 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:07.798298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:07.798335 (+ 37us) server_negotiation.cc:407] Connection header received
0621 05:25:07.798981 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:07.798989 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:07.799049 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:07.799204 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:07.799898 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:07.801557 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:07.802547 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:07.803092 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:07.803692 (+ 600us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:07.806633 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:07.806637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:07.806638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:07.806649 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:07.808646 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:07.809110 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:07.809112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:07.809113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:07.809211 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:07.809451 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:07.809452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:07.809452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:07.809751 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:07.810283 (+ 532us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:07.810789 (+ 506us) server_negotiation.cc:299] Negotiation successful
0621 05:25:07.810888 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":108,"threads_started":1}
I20260621 05:25:07.811995 23180 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:25:07.812145 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:25:07.812526 23180 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:25:07.820320 23180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c97f6ae0cf484807aec50b8ff30e9852: Generated new TSK 0
I20260621 05:25:07.820475 23180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:25:07.822008 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:07.813652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:52403 (local address 127.0.0.1:44759)
0621 05:25:07.813688 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:07.813689 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:07.813747 (+ 58us) server_negotiation.cc:407] Connection header received
0621 05:25:07.813841 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:07.813842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:07.813851 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:07.813879 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:07.814529 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:07.815346 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:07.816673 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:07.817010 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:07.817088 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:07.819078 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:07.819080 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:07.819080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:07.819109 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:07.820812 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:07.821204 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:07.821205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:07.821205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:07.821261 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:07.821506 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:07.821507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:07.821508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:07.821617 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:07.821733 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:07.821853 (+ 120us) server_negotiation.cc:299] Negotiation successful
0621 05:25:07.821915 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:25:07.824780 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 23090
W20260621 05:25:07.829216 23075 connection.cc:535] client connection to 127.0.0.1:44759 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44759): Connection reset by peer (error 104)
2026-06-21T05:25:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:25:07 dist-test-slave-b357 krb5kdc[23189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:25:07 dist-test-slave-b357 krb5kdc[23189](info): set up 2 sockets
Jun 21 05:25:07 dist-test-slave-b357 krb5kdc[23189](info): commencing operation
krb5kdc: starting...
W20260621 05:25:08.891295 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:25:08 dist-test-slave-b357 krb5kdc[23189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:25:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:25:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:25:09.016085 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:25:09.104151 23202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:09.104421 23202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:09.104465 23202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:09.106130 23202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:09.106184 23202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:09.106200 23202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:09.106213 23202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:09.106230 23202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:09.107895 23202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:09.108132 23202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:09.108309 23202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:09.113751 23210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:09.113725 23207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:09.113770 23208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:09.113828 23202 server_base.cc:1034] running on GCE node
I20260621 05:25:09.114205 23202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:09.114493 23202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:09.115680 23202 hybrid_clock.cc:648] HybridClock initialized: now 1782019509115653 us; error 49 us; skew 500 ppm
Jun 21 05:25:09 dist-test-slave-b357 krb5kdc[23189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:09.118841 23202 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:25:09.119443 23202 webserver.cc:460] Webserver started at http://127.0.0.1:39777/ using document root <none> and password file <none>
I20260621 05:25:09.119684 23202 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:09.119756 23202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:09.119920 23202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:09.120994 23202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "e8063b9628ee4b3d9dbf27325525b4e4"
format_stamp: "Formatted at 2026-06-21 05:25:09 on dist-test-slave-b357"
server_key: "55311521d92078cbe5f6df1e2333cfe3"
server_key_iv: "3c89d514c12f655e0aef073d5405b35d"
server_key_version: "encryptionkey@0"
I20260621 05:25:09.121367 23202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "e8063b9628ee4b3d9dbf27325525b4e4"
format_stamp: "Formatted at 2026-06-21 05:25:09 on dist-test-slave-b357"
server_key: "55311521d92078cbe5f6df1e2333cfe3"
server_key_iv: "3c89d514c12f655e0aef073d5405b35d"
server_key_version: "encryptionkey@0"
I20260621 05:25:09.123063 23202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:25:09.124402 23217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:09.124642 23202 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:25:09.124771 23202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "e8063b9628ee4b3d9dbf27325525b4e4"
format_stamp: "Formatted at 2026-06-21 05:25:09 on dist-test-slave-b357"
server_key: "55311521d92078cbe5f6df1e2333cfe3"
server_key_iv: "3c89d514c12f655e0aef073d5405b35d"
server_key_version: "encryptionkey@0"
I20260621 05:25:09.124893 23202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:09.137971 23202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:09.140443 23202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:09.140617 23202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:09.145031 23202 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33877
I20260621 05:25:09.145062 23269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33877 every 8 connection(s)
I20260621 05:25:09.145404 23202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:25:09.146332 23270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:09.148605 23270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4: Bootstrap starting.
I20260621 05:25:09.149200 23270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:09.149492 23270 log.cc:824] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:09.150573 23270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4: No bootstrap required, opened a new log
I20260621 05:25:09.150962 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 23202
I20260621 05:25:09.151106 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:25:09.151495 2176 external_mini_cluster.cc:1428] Setting key 7f1b3f0bf30a52e1cfdcf5340919e5c9
I20260621 05:25:09.153246 23270 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8063b9628ee4b3d9dbf27325525b4e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33877 } }
I20260621 05:25:09.153417 23270 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:25:09.153441 23270 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8063b9628ee4b3d9dbf27325525b4e4, State: Initialized, Role: FOLLOWER
I20260621 05:25:09.153649 23270 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8063b9628ee4b3d9dbf27325525b4e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33877 } }
I20260621 05:25:09.153761 23270 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:25:09.153797 23270 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:25:09.153870 23270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:09.154623 23270 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8063b9628ee4b3d9dbf27325525b4e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33877 } }
I20260621 05:25:09.154765 23270 leader_election.cc:302] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8063b9628ee4b3d9dbf27325525b4e4; no voters:
I20260621 05:25:09.154985 23270 leader_election.cc:288] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:25:09.155012 23275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:25:09.155184 23275 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4 [term 1 LEADER]: Becoming Leader. State: Replica: e8063b9628ee4b3d9dbf27325525b4e4, State: Running, Role: LEADER
I20260621 05:25:09.155350 23270 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:09.155350 23275 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8063b9628ee4b3d9dbf27325525b4e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33877 } }
I20260621 05:25:09.155925 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8063b9628ee4b3d9dbf27325525b4e4. Latest consensus state: current_term: 1 leader_uuid: "e8063b9628ee4b3d9dbf27325525b4e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8063b9628ee4b3d9dbf27325525b4e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33877 } } }
I20260621 05:25:09.156029 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:09.156275 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8063b9628ee4b3d9dbf27325525b4e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8063b9628ee4b3d9dbf27325525b4e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33877 } } }
I20260621 05:25:09.156682 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:09.156888 23283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:25:09.157559 23283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:25:09.158998 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:09.152267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:55355 (local address 127.0.0.1:33877)
0621 05:25:09.152496 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:09.152520 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:09.152559 (+ 39us) server_negotiation.cc:407] Connection header received
0621 05:25:09.153283 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:09.153294 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:09.153356 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:09.153563 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:09.153936 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:09.155371 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:09.156455 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:09.157078 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:09.157268 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:09.158103 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:09.158109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:09.158109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:09.158384 (+ 275us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:09.158424 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:09.158436 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:09.158556 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:09.158682 (+ 126us) server_negotiation.cc:299] Negotiation successful
0621 05:25:09.158716 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":111,"threads_started":1}
I20260621 05:25:09.159485 23283 catalog_manager.cc:1347] Generated new cluster ID: ca22e93c4d3544129d457eb63aca6d9c
I20260621 05:25:09.159621 23283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:25:09.160322 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:25:09.166426 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:09.163490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:34941 (local address 127.0.0.1:33877)
0621 05:25:09.163530 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:09.163533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:09.163760 (+ 227us) server_negotiation.cc:407] Connection header received
0621 05:25:09.163818 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:09.163819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:09.163830 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:09.163886 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:09.164007 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:09.164475 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:09.165170 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:09.165407 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:09.165484 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:09.166171 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:09.166172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:09.166173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:09.166178 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:09.166193 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:09.166198 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:09.166246 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:09.166316 (+ 70us) server_negotiation.cc:299] Negotiation successful
0621 05:25:09.166328 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:25:09.170451 23283 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:25:09.171101 23283 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:25:09.176555 23283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8063b9628ee4b3d9dbf27325525b4e4: Generated new TSK 0
I20260621 05:25:09.176714 23283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:25:09.183869 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 23202
W20260621 05:25:09.188221 23187 connection.cc:535] client connection to 127.0.0.1:33877 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33877): Connection reset by peer (error 104)
2026-06-21T05:25:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:25:09 dist-test-slave-b357 krb5kdc[23301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:25:09 dist-test-slave-b357 krb5kdc[23301](info): set up 2 sockets
Jun 21 05:25:09 dist-test-slave-b357 krb5kdc[23301](info): commencing operation
krb5kdc: starting...
W20260621 05:25:10.239952 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:25:10 dist-test-slave-b357 krb5kdc[23301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:25:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:25:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:25:10.362517 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38261
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:25:10.447329 23314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:10.447558 23314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:10.447595 23314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:10.448719 23314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:10.448796 23314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:10.448823 23314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:10.448853 23314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:10.448876 23314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:10.450299 23314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:10.450555 23314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:10.450773 23314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:10.456007 23320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:10.455988 23319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:10.455976 23322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:10.456427 23314 server_base.cc:1034] running on GCE node
I20260621 05:25:10.456686 23314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:10.456990 23314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:10.458158 23314 hybrid_clock.cc:648] HybridClock initialized: now 1782019510458130 us; error 46 us; skew 500 ppm
Jun 21 05:25:10 dist-test-slave-b357 krb5kdc[23301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:10.461345 23314 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:25:10.461879 23314 webserver.cc:460] Webserver started at http://127.0.0.1:41963/ using document root <none> and password file <none>
I20260621 05:25:10.462102 23314 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:10.462147 23314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:10.462257 23314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:10.463384 23314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "7f49756faab64ff99e62f961baf965fb"
format_stamp: "Formatted at 2026-06-21 05:25:10 on dist-test-slave-b357"
server_key: "68dc0021c48c14d7f80bfb4ce8ebece1"
server_key_iv: "fcf999b936f3f86a10aec0e9f977a6ae"
server_key_version: "encryptionkey@0"
I20260621 05:25:10.463738 23314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "7f49756faab64ff99e62f961baf965fb"
format_stamp: "Formatted at 2026-06-21 05:25:10 on dist-test-slave-b357"
server_key: "68dc0021c48c14d7f80bfb4ce8ebece1"
server_key_iv: "fcf999b936f3f86a10aec0e9f977a6ae"
server_key_version: "encryptionkey@0"
I20260621 05:25:10.465044 23314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:25:10.466013 23329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:10.466322 23314 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:25:10.466507 23314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "7f49756faab64ff99e62f961baf965fb"
format_stamp: "Formatted at 2026-06-21 05:25:10 on dist-test-slave-b357"
server_key: "68dc0021c48c14d7f80bfb4ce8ebece1"
server_key_iv: "fcf999b936f3f86a10aec0e9f977a6ae"
server_key_version: "encryptionkey@0"
I20260621 05:25:10.466598 23314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:10.469902 23314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:10.472675 23314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:10.472854 23314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:10.477207 23314 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38261
I20260621 05:25:10.477238 23381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38261 every 8 connection(s)
I20260621 05:25:10.477546 23314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:25:10.477979 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 23314
I20260621 05:25:10.478097 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:25:10.478271 2176 external_mini_cluster.cc:1428] Setting key 42f62a0beea63efdd221d166c2c1c6cb
I20260621 05:25:10.478672 23382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:10.480783 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:10.478766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:34325 (local address 127.0.0.1:38261)
0621 05:25:10.478952 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:10.478959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:10.478981 (+ 22us) server_negotiation.cc:407] Connection header received
0621 05:25:10.479452 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:10.479460 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:10.479491 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:10.479664 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:10.480355 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:10.480358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:10.480358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:10.480362 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:10.480405 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:10.480415 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:10.480489 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:10.480554 (+ 65us) server_negotiation.cc:299] Negotiation successful
0621 05:25:10.480595 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260621 05:25:10.481767 23382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb: Bootstrap starting.
I20260621 05:25:10.482018 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:25:10.482482 23382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:10.482829 23382 log.cc:824] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:10.483624 23382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb: No bootstrap required, opened a new log
I20260621 05:25:10.484607 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:10.483595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:44547 (local address 127.0.0.1:38261)
0621 05:25:10.483635 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:10.483637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:10.483725 (+ 88us) server_negotiation.cc:407] Connection header received
0621 05:25:10.483806 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:10.483807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:10.483810 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:10.483852 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:10.484375 (+ 523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:10.484376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:10.484377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:10.484380 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:10.484392 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:10.484395 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:10.484445 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:10.484512 (+ 67us) server_negotiation.cc:299] Negotiation successful
0621 05:25:10.484522 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 05:25:10.485749 23382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f49756faab64ff99e62f961baf965fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38261 } }
I20260621 05:25:10.485908 23382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:25:10.485954 23382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f49756faab64ff99e62f961baf965fb, State: Initialized, Role: FOLLOWER
I20260621 05:25:10.486066 23382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f49756faab64ff99e62f961baf965fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38261 } }
I20260621 05:25:10.486163 23382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:25:10.486205 23382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:25:10.486254 23382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:10.486784 23382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f49756faab64ff99e62f961baf965fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38261 } }
I20260621 05:25:10.486878 23382 leader_election.cc:302] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f49756faab64ff99e62f961baf965fb; no voters:
I20260621 05:25:10.487085 23382 leader_election.cc:288] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:25:10.487226 23392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:25:10.487341 23382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:10.487388 23392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb [term 1 LEADER]: Becoming Leader. State: Replica: 7f49756faab64ff99e62f961baf965fb, State: Running, Role: LEADER
I20260621 05:25:10.487511 23392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f49756faab64ff99e62f961baf965fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38261 } }
I20260621 05:25:10.487936 23394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f49756faab64ff99e62f961baf965fb. Latest consensus state: current_term: 1 leader_uuid: "7f49756faab64ff99e62f961baf965fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f49756faab64ff99e62f961baf965fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38261 } } }
I20260621 05:25:10.488044 23394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:10.488270 23401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:25:10.488358 23393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f49756faab64ff99e62f961baf965fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f49756faab64ff99e62f961baf965fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38261 } } }
I20260621 05:25:10.488430 23393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:10.488719 23401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:25:10.490620 23401 catalog_manager.cc:1347] Generated new cluster ID: 394128a0fd124a7a87d1400f656d63f0
I20260621 05:25:10.490723 23401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:25:10.509619 23401 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:25:10.509923 23401 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:25:10.524664 23401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f49756faab64ff99e62f961baf965fb: Generated new TSK 0
I20260621 05:25:10.524868 23401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:25:10.532311 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 23314
2026-06-21T05:25:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:25:10 dist-test-slave-b357 krb5kdc[23413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:25:10 dist-test-slave-b357 krb5kdc[23413](info): set up 2 sockets
Jun 21 05:25:10 dist-test-slave-b357 krb5kdc[23413](info): commencing operation
krb5kdc: starting...
W20260621 05:25:11.595441 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:25:11 dist-test-slave-b357 krb5kdc[23413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:25:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:25:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:25:11.725898 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32993
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:25:11.817214 23426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:11.817493 23426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:11.817523 23426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:11.818825 23426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:11.818890 23426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:11.818909 23426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:11.818923 23426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:11.818936 23426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:11.820786 23426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32993
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:11.821117 23426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:11.821431 23426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:11.827373 23434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:11.827390 23432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:11.827385 23431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:11.827890 23426 server_base.cc:1034] running on GCE node
I20260621 05:25:11.828158 23426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:11.828445 23426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:11.829605 23426 hybrid_clock.cc:648] HybridClock initialized: now 1782019511829589 us; error 53 us; skew 500 ppm
Jun 21 05:25:11 dist-test-slave-b357 krb5kdc[23413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:11.836481 23426 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:25:11.837014 23426 webserver.cc:460] Webserver started at http://127.0.0.1:45343/ using document root <none> and password file <none>
I20260621 05:25:11.837303 23426 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:11.837378 23426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:11.837639 23426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:11.838897 23426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "2dfc13ffb35e494f80d47eddb1f9e184"
format_stamp: "Formatted at 2026-06-21 05:25:11 on dist-test-slave-b357"
server_key: "a29c21bb9adb4262d3eb56655d3e4059"
server_key_iv: "318d6477edcb02ecaba4b1fd2924ac2a"
server_key_version: "encryptionkey@0"
I20260621 05:25:11.839301 23426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "2dfc13ffb35e494f80d47eddb1f9e184"
format_stamp: "Formatted at 2026-06-21 05:25:11 on dist-test-slave-b357"
server_key: "a29c21bb9adb4262d3eb56655d3e4059"
server_key_iv: "318d6477edcb02ecaba4b1fd2924ac2a"
server_key_version: "encryptionkey@0"
I20260621 05:25:11.840953 23426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:25:11.842012 23441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:11.842316 23426 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:25:11.842433 23426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "2dfc13ffb35e494f80d47eddb1f9e184"
format_stamp: "Formatted at 2026-06-21 05:25:11 on dist-test-slave-b357"
server_key: "a29c21bb9adb4262d3eb56655d3e4059"
server_key_iv: "318d6477edcb02ecaba4b1fd2924ac2a"
server_key_version: "encryptionkey@0"
I20260621 05:25:11.842519 23426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:11.863438 23426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:11.865899 23426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:11.866199 23426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:11.870362 23426 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32993
I20260621 05:25:11.870414 23493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32993 every 8 connection(s)
I20260621 05:25:11.870787 23426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:25:11.871371 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 23426
I20260621 05:25:11.871498 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:25:11.871709 2176 external_mini_cluster.cc:1428] Setting key 88b60b91b0f16848f9c17c4f77146a73
I20260621 05:25:11.872116 23494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:11.875581 23494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184: Bootstrap starting.
I20260621 05:25:11.876194 23494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:11.876482 23494 log.cc:824] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:11.877125 23494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184: No bootstrap required, opened a new log
I20260621 05:25:11.878839 23494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfc13ffb35e494f80d47eddb1f9e184" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32993 } }
I20260621 05:25:11.878993 23494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:25:11.879043 23494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dfc13ffb35e494f80d47eddb1f9e184, State: Initialized, Role: FOLLOWER
Jun 21 05:25:11 dist-test-slave-b357 krb5kdc[23413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 05:25:11.879215 23494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfc13ffb35e494f80d47eddb1f9e184" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32993 } }
I20260621 05:25:11.879295 23494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:25:11.879317 23494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:25:11.879345 23494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:11.879884 23494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfc13ffb35e494f80d47eddb1f9e184" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32993 } }
I20260621 05:25:11.880005 23494 leader_election.cc:302] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dfc13ffb35e494f80d47eddb1f9e184; no voters:
I20260621 05:25:11.880195 23494 leader_election.cc:288] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:25:11.880225 23499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:25:11.880358 23499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184 [term 1 LEADER]: Becoming Leader. State: Replica: 2dfc13ffb35e494f80d47eddb1f9e184, State: Running, Role: LEADER
I20260621 05:25:11.880462 23494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:11.880564 23499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfc13ffb35e494f80d47eddb1f9e184" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32993 } }
I20260621 05:25:11.880937 23499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dfc13ffb35e494f80d47eddb1f9e184. Latest consensus state: current_term: 1 leader_uuid: "2dfc13ffb35e494f80d47eddb1f9e184" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfc13ffb35e494f80d47eddb1f9e184" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32993 } } }
I20260621 05:25:11.881073 23499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:11.881592 23501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dfc13ffb35e494f80d47eddb1f9e184" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfc13ffb35e494f80d47eddb1f9e184" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32993 } } }
I20260621 05:25:11.881680 23501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184 [sys.catalog]: This master's current role is: LEADER
W20260621 05:25:11.882987 23515 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:25:11.883050 23515 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:25:11.883169 23505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:25:11.883333 23505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:25:11.884832 23505 catalog_manager.cc:1347] Generated new cluster ID: 9de208318f0d40a5ad5eb6ca205af582
I20260621 05:25:11.884881 23505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:25:11.885097 23497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:11.872272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:34333 (local address 127.0.0.1:32993)
0621 05:25:11.872485 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:11.872493 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:11.872524 (+ 31us) server_negotiation.cc:407] Connection header received
0621 05:25:11.873244 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:11.873252 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:11.873322 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:11.873473 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:11.874143 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:11.875522 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:11.876388 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:11.876845 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:11.876979 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:11.879947 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:11.879953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:11.879953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:11.879968 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:11.882686 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:11.883283 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:11.883286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:11.883286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:11.883401 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:11.883807 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:11.883808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:11.883808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:11.884106 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:11.884712 (+ 606us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:11.884798 (+ 86us) server_negotiation.cc:299] Negotiation successful
0621 05:25:11.884906 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":101,"threads_started":1}
I20260621 05:25:11.886098 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:25:11.894886 23497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:11.887636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:48531 (local address 127.0.0.1:32993)
0621 05:25:11.887669 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:11.887671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:11.887943 (+ 272us) server_negotiation.cc:407] Connection header received
0621 05:25:11.887982 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:11.887983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:11.887992 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:11.888047 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:11.888628 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:11.889412 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:11.890365 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:11.890697 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:11.890771 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:11.892125 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:11.892127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:11.892127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:11.892134 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:11.893694 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:11.894080 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:11.894082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:11.894082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:11.894141 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:11.894367 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:11.894368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:11.894368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:11.894491 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:11.894589 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:11.894735 (+ 146us) server_negotiation.cc:299] Negotiation successful
0621 05:25:11.894801 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260621 05:25:11.904150 23505 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:25:11.904767 23505 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:25:11.908941 23505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dfc13ffb35e494f80d47eddb1f9e184: Generated new TSK 0
I20260621 05:25:11.909107 23505 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:25:11.912324 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 23426
W20260621 05:25:11.917325 23411 connection.cc:535] client connection to 127.0.0.1:32993 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32993): Connection reset by peer (error 104)
2026-06-21T05:25:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:25:11 dist-test-slave-b357 krb5kdc[23525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:25:11 dist-test-slave-b357 krb5kdc[23525](info): set up 2 sockets
Jun 21 05:25:11 dist-test-slave-b357 krb5kdc[23525](info): commencing operation
krb5kdc: starting...
W20260621 05:25:12.975821 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:25:13 dist-test-slave-b357 krb5kdc[23525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:25:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:25:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:25:13.102317 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38717
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:25:13.183413 23538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:13.183648 23538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:13.183684 23538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:13.184815 23538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:13.184880 23538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:13.184906 23538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:13.184937 23538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:13.184960 23538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:13.186389 23538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38717
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:13.186650 23538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:13.186864 23538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:13.191895 23546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:13.192028 23544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:13.192026 23543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:13.192207 23538 server_base.cc:1034] running on GCE node
I20260621 05:25:13.192493 23538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:13.192775 23538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:13.193955 23538 hybrid_clock.cc:648] HybridClock initialized: now 1782019513193971 us; error 119 us; skew 500 ppm
Jun 21 05:25:13 dist-test-slave-b357 krb5kdc[23525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:13.196746 23538 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:25:13.197127 23538 webserver.cc:460] Webserver started at http://127.0.0.1:36157/ using document root <none> and password file <none>
I20260621 05:25:13.197358 23538 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:13.197405 23538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:13.197594 23538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:13.198659 23538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "d33531a356f74d13b40597aac8232691"
format_stamp: "Formatted at 2026-06-21 05:25:13 on dist-test-slave-b357"
server_key: "1efd113189e61647c5bf5678146be02f"
server_key_iv: "95c8e5507af5a4e1c7a4087be52175e4"
server_key_version: "encryptionkey@0"
I20260621 05:25:13.199024 23538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "d33531a356f74d13b40597aac8232691"
format_stamp: "Formatted at 2026-06-21 05:25:13 on dist-test-slave-b357"
server_key: "1efd113189e61647c5bf5678146be02f"
server_key_iv: "95c8e5507af5a4e1c7a4087be52175e4"
server_key_version: "encryptionkey@0"
I20260621 05:25:13.200646 23538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:25:13.201861 23553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:13.202242 23538 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260621 05:25:13.202356 23538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "d33531a356f74d13b40597aac8232691"
format_stamp: "Formatted at 2026-06-21 05:25:13 on dist-test-slave-b357"
server_key: "1efd113189e61647c5bf5678146be02f"
server_key_iv: "95c8e5507af5a4e1c7a4087be52175e4"
server_key_version: "encryptionkey@0"
I20260621 05:25:13.202443 23538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:13.211962 23538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:13.214318 23538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:13.214488 23538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:13.220559 23538 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38717
I20260621 05:25:13.220711 23605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38717 every 8 connection(s)
I20260621 05:25:13.221057 23538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:25:13.221997 23606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:13.224225 23606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691: Bootstrap starting.
I20260621 05:25:13.224786 23606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:13.225095 23606 log.cc:824] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:13.225857 23606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691: No bootstrap required, opened a new log
I20260621 05:25:13.227195 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 23538
I20260621 05:25:13.227347 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:25:13.227524 2176 external_mini_cluster.cc:1428] Setting key 34d73b1ba3cc3c6def957c523e41ca05
I20260621 05:25:13.227626 23606 raft_consensus.cc:348] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33531a356f74d13b40597aac8232691" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38717 } }
I20260621 05:25:13.227805 23606 raft_consensus.cc:374] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:25:13.228017 23606 raft_consensus.cc:729] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33531a356f74d13b40597aac8232691, State: Initialized, Role: FOLLOWER
I20260621 05:25:13.228432 23606 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33531a356f74d13b40597aac8232691" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38717 } }
I20260621 05:25:13.228581 23606 raft_consensus.cc:388] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:25:13.228647 23606 raft_consensus.cc:482] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:25:13.228710 23606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:13.229737 23606 raft_consensus.cc:504] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33531a356f74d13b40597aac8232691" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38717 } }
I20260621 05:25:13.229868 23606 leader_election.cc:302] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d33531a356f74d13b40597aac8232691; no voters:
I20260621 05:25:13.230053 23606 leader_election.cc:288] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:25:13.230149 23611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:25:13.230362 23611 raft_consensus.cc:686] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691 [term 1 LEADER]: Becoming Leader. State: Replica: d33531a356f74d13b40597aac8232691, State: Running, Role: LEADER
I20260621 05:25:13.230392 23606 sys_catalog.cc:564] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:13.230484 23611 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33531a356f74d13b40597aac8232691" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38717 } }
I20260621 05:25:13.230876 23611 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33531a356f74d13b40597aac8232691. Latest consensus state: current_term: 1 leader_uuid: "d33531a356f74d13b40597aac8232691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33531a356f74d13b40597aac8232691" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38717 } } }
I20260621 05:25:13.230890 23612 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d33531a356f74d13b40597aac8232691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33531a356f74d13b40597aac8232691" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38717 } } }
I20260621 05:25:13.231043 23612 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:13.231035 23611 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:13.231446 23620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:25:13.231992 23620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:25:13.233868 23620 catalog_manager.cc:1347] Generated new cluster ID: 9094b765d4b84d8eabbbec93c7701247
I20260621 05:25:13.233930 23620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:25:13.234615 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:13.228032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:42101 (local address 127.0.0.1:38717)
0621 05:25:13.228187 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:13.228198 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:13.228242 (+ 44us) server_negotiation.cc:407] Connection header received
0621 05:25:13.228429 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:13.228435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:13.228478 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:13.228624 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:13.228931 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:13.230743 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:13.232122 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:13.232776 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:13.232919 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:13.233702 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:13.233706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:13.233707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:13.234022 (+ 315us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:13.234077 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:13.234091 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:13.234188 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:13.234270 (+ 82us) server_negotiation.cc:299] Negotiation successful
0621 05:25:13.234301 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260621 05:25:13.236145 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:25:13.241080 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:13.237621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:39993 (local address 127.0.0.1:38717)
0621 05:25:13.237663 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:13.237665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:13.237724 (+ 59us) server_negotiation.cc:407] Connection header received
0621 05:25:13.237800 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:13.237801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:13.237810 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:13.237882 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:13.238016 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:13.238789 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:13.239767 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:13.240077 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:13.240154 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:13.240822 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:13.240824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:13.240824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:13.240829 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:13.240845 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:13.240851 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:13.240918 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:13.240988 (+ 70us) server_negotiation.cc:299] Negotiation successful
0621 05:25:13.241002 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:25:13.264190 23620 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:25:13.264931 23620 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:25:13.274075 23620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d33531a356f74d13b40597aac8232691: Generated new TSK 0
I20260621 05:25:13.274278 23620 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:25:13.280854 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 23538
W20260621 05:25:13.286044 23523 connection.cc:535] client connection to 127.0.0.1:38717 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38717): Connection reset by peer (error 104)
2026-06-21T05:25:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:25:13 dist-test-slave-b357 krb5kdc[23637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:25:13 dist-test-slave-b357 krb5kdc[23637](info): set up 2 sockets
Jun 21 05:25:13 dist-test-slave-b357 krb5kdc[23637](info): commencing operation
krb5kdc: starting...
W20260621 05:25:14.371493 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:25:14 dist-test-slave-b357 krb5kdc[23637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:25:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:25:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 05:25:14.502606 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44459
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:25:14.595464 23650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:14.595744 23650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:14.595782 23650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:14.598733 23650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:14.598852 23650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:14.598881 23650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:14.598906 23650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:14.598937 23650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:14.601173 23650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44459
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:14.601485 23650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:14.601739 23650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:14.606786 23658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:14.606786 23656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:14.606832 23655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:14.607067 23650 server_base.cc:1034] running on GCE node
I20260621 05:25:14.607448 23650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:14.607760 23650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:14.608968 23650 hybrid_clock.cc:648] HybridClock initialized: now 1782019514608941 us; error 42 us; skew 500 ppm
Jun 21 05:25:14 dist-test-slave-b357 krb5kdc[23637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:14.611903 23650 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 05:25:14.612296 23650 webserver.cc:460] Webserver started at http://127.0.0.1:35287/ using document root <none> and password file <none>
I20260621 05:25:14.612475 23650 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:14.612516 23650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:14.612612 23650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:14.613634 23650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "1b31877015b042e3b9a6e83ef5fd4d89"
format_stamp: "Formatted at 2026-06-21 05:25:14 on dist-test-slave-b357"
server_key: "d645475f334253886774a2b201a74348"
server_key_iv: "e75bf487cadefac70b44f1eeed8a1cdb"
server_key_version: "encryptionkey@0"
I20260621 05:25:14.614076 23650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "1b31877015b042e3b9a6e83ef5fd4d89"
format_stamp: "Formatted at 2026-06-21 05:25:14 on dist-test-slave-b357"
server_key: "d645475f334253886774a2b201a74348"
server_key_iv: "e75bf487cadefac70b44f1eeed8a1cdb"
server_key_version: "encryptionkey@0"
I20260621 05:25:14.615553 23650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260621 05:25:14.616401 23665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:14.616662 23650 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:25:14.616750 23650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "1b31877015b042e3b9a6e83ef5fd4d89"
format_stamp: "Formatted at 2026-06-21 05:25:14 on dist-test-slave-b357"
server_key: "d645475f334253886774a2b201a74348"
server_key_iv: "e75bf487cadefac70b44f1eeed8a1cdb"
server_key_version: "encryptionkey@0"
I20260621 05:25:14.616811 23650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:14.620041 23650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:14.622506 23650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:14.622691 23650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:14.627081 23650 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44459
I20260621 05:25:14.627172 23717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44459 every 8 connection(s)
I20260621 05:25:14.627490 23650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:25:14.627624 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 23650
I20260621 05:25:14.627759 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:25:14.627947 2176 external_mini_cluster.cc:1428] Setting key fc6f6d75196879a24d5e88982b8d6962
I20260621 05:25:14.628679 23718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:14.630996 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:14.628494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:36767 (local address 127.0.0.1:44459)
0621 05:25:14.628694 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:14.628703 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:14.628736 (+ 33us) server_negotiation.cc:407] Connection header received
0621 05:25:14.629304 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:14.629312 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:14.629342 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:14.629487 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:14.630450 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:14.630453 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:14.630454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:14.630458 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:14.630502 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:14.630516 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:14.630614 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:14.630773 (+ 159us) server_negotiation.cc:299] Negotiation successful
0621 05:25:14.630816 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
I20260621 05:25:14.631767 23718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89: Bootstrap starting.
I20260621 05:25:14.632037 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:25:14.632444 23718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:14.632733 23718 log.cc:824] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:14.633404 23718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89: No bootstrap required, opened a new log
I20260621 05:25:14.635090 23718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b31877015b042e3b9a6e83ef5fd4d89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44459 } }
I20260621 05:25:14.635277 23718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:25:14.635303 23718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b31877015b042e3b9a6e83ef5fd4d89, State: Initialized, Role: FOLLOWER
I20260621 05:25:14.635437 23718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b31877015b042e3b9a6e83ef5fd4d89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44459 } }
I20260621 05:25:14.635519 23718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:25:14.635540 23718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:25:14.635591 23718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:14.636101 23718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b31877015b042e3b9a6e83ef5fd4d89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44459 } }
I20260621 05:25:14.636183 23718 leader_election.cc:302] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b31877015b042e3b9a6e83ef5fd4d89; no voters:
I20260621 05:25:14.636350 23718 leader_election.cc:288] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:25:14.636454 23727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:25:14.636569 23718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:14.636610 23727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89 [term 1 LEADER]: Becoming Leader. State: Replica: 1b31877015b042e3b9a6e83ef5fd4d89, State: Running, Role: LEADER
I20260621 05:25:14.636703 23727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b31877015b042e3b9a6e83ef5fd4d89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44459 } }
I20260621 05:25:14.637054 23729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b31877015b042e3b9a6e83ef5fd4d89. Latest consensus state: current_term: 1 leader_uuid: "1b31877015b042e3b9a6e83ef5fd4d89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b31877015b042e3b9a6e83ef5fd4d89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44459 } } }
I20260621 05:25:14.637164 23729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:14.637331 23728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b31877015b042e3b9a6e83ef5fd4d89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b31877015b042e3b9a6e83ef5fd4d89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44459 } } }
I20260621 05:25:14.637398 23728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:14.637714 23736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:25:14.638180 23736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:25:14.639655 23736 catalog_manager.cc:1347] Generated new cluster ID: d3a1aa22cd484a679c9c650c9034877e
I20260621 05:25:14.639703 23736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:25:14.640255 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:14.638985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:51161 (local address 127.0.0.1:44459)
0621 05:25:14.639022 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:14.639024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:14.639237 (+ 213us) server_negotiation.cc:407] Connection header received
0621 05:25:14.639309 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:14.639309 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:14.639314 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:14.639364 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:14.640034 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:14.640035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:14.640035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:14.640040 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:14.640055 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:14.640059 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:14.640125 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:14.640181 (+ 56us) server_negotiation.cc:299] Negotiation successful
0621 05:25:14.640194 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:25:14.645298 23736 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:25:14.645605 23736 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:25:14.652311 23736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b31877015b042e3b9a6e83ef5fd4d89: Generated new TSK 0
I20260621 05:25:14.652506 23736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:25:14.654734 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 23650
2026-06-21T05:25:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:25:14 dist-test-slave-b357 krb5kdc[23749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:25:14 dist-test-slave-b357 krb5kdc[23749](info): set up 2 sockets
Jun 21 05:25:14 dist-test-slave-b357 krb5kdc[23749](info): commencing operation
krb5kdc: starting...
W20260621 05:25:15.712396 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:25:15 dist-test-slave-b357 krb5kdc[23749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:25:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:25:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:25:15.838968 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:34975
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:34975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:25:15.918263 23762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:15.918462 23762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:15.918484 23762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:15.919636 23762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:15.919675 23762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:15.919689 23762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:15.919703 23762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:15.919715 23762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:15.921164 23762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:34975
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:34975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:15.921399 23762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:15.921622 23762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:15.927170 23768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:15.927198 23767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:15.927155 23770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:15.927436 23762 server_base.cc:1034] running on GCE node
I20260621 05:25:15.927685 23762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:15.928090 23762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:15.929361 23762 hybrid_clock.cc:648] HybridClock initialized: now 1782019515929324 us; error 51 us; skew 500 ppm
Jun 21 05:25:15 dist-test-slave-b357 krb5kdc[23749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019515, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:15.932636 23762 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:25:15.933157 23762 webserver.cc:460] Webserver started at http://127.2.32.62:45321/ using document root <none> and password file <none>
I20260621 05:25:15.933425 23762 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:15.933497 23762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:15.933648 23762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:15.934852 23762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "6b8da8b5e326404588e992abe11ccffb"
format_stamp: "Formatted at 2026-06-21 05:25:15 on dist-test-slave-b357"
server_key: "7d959435e9b8620cb32adc522109c894"
server_key_iv: "6578bb87e058c035e4ff740b5098e375"
server_key_version: "encryptionkey@0"
I20260621 05:25:15.935297 23762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "6b8da8b5e326404588e992abe11ccffb"
format_stamp: "Formatted at 2026-06-21 05:25:15 on dist-test-slave-b357"
server_key: "7d959435e9b8620cb32adc522109c894"
server_key_iv: "6578bb87e058c035e4ff740b5098e375"
server_key_version: "encryptionkey@0"
I20260621 05:25:15.936724 23762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:25:15.937603 23777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:15.937849 23762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:25:15.937963 23762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "6b8da8b5e326404588e992abe11ccffb"
format_stamp: "Formatted at 2026-06-21 05:25:15 on dist-test-slave-b357"
server_key: "7d959435e9b8620cb32adc522109c894"
server_key_iv: "6578bb87e058c035e4ff740b5098e375"
server_key_version: "encryptionkey@0"
I20260621 05:25:15.938050 23762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:15.966365 23762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:15.969177 23762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:15.969354 23762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:15.974661 23762 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:34975
I20260621 05:25:15.974706 23829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:34975 every 8 connection(s)
I20260621 05:25:15.975239 23762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:25:15.976336 23830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:15.979593 23830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb: Bootstrap starting.
I20260621 05:25:15.980193 23830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:15.980486 23830 log.cc:824] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:15.981240 23830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb: No bootstrap required, opened a new log
I20260621 05:25:15.982895 23830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8da8b5e326404588e992abe11ccffb" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34975 } }
I20260621 05:25:15.983034 23830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:25:15.983057 23830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8da8b5e326404588e992abe11ccffb, State: Initialized, Role: FOLLOWER
I20260621 05:25:15.983337 23830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8da8b5e326404588e992abe11ccffb" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34975 } }
I20260621 05:25:15.983429 23830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:25:15.983480 23830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:25:15.983533 23830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:15.983980 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 23762
I20260621 05:25:15.984123 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:25:15.984306 2176 external_mini_cluster.cc:1428] Setting key 57bfbe1fc39248269900f6780b23e2be
I20260621 05:25:15.984593 23830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8da8b5e326404588e992abe11ccffb" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34975 } }
I20260621 05:25:15.984728 23830 leader_election.cc:302] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b8da8b5e326404588e992abe11ccffb; no voters:
I20260621 05:25:15.984891 23830 leader_election.cc:288] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:25:15.984983 23834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:25:15.985195 23830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:15.985240 23834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb [term 1 LEADER]: Becoming Leader. State: Replica: 6b8da8b5e326404588e992abe11ccffb, State: Running, Role: LEADER
I20260621 05:25:15.985412 23834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8da8b5e326404588e992abe11ccffb" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34975 } }
I20260621 05:25:15.985899 23835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b8da8b5e326404588e992abe11ccffb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8da8b5e326404588e992abe11ccffb" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34975 } } }
I20260621 05:25:15.985998 23835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:15.986397 23837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b8da8b5e326404588e992abe11ccffb. Latest consensus state: current_term: 1 leader_uuid: "6b8da8b5e326404588e992abe11ccffb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b8da8b5e326404588e992abe11ccffb" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 34975 } } }
I20260621 05:25:15.986474 23837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb [sys.catalog]: This master's current role is: LEADER
W20260621 05:25:15.987179 23850 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:25:15.987258 23850 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:25:15.987303 23844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:25:15.987450 23844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:25:15.988945 23844 catalog_manager.cc:1347] Generated new cluster ID: 25c4da73e2694fb19ba414095db5156a
I20260621 05:25:15.988997 23844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:25:15 dist-test-slave-b357 krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:25:15.997506 23836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:15.984961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41439 (local address 127.2.32.62:34975)
0621 05:25:15.985141 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:15.985150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:15.986088 (+ 938us) server_negotiation.cc:407] Connection header received
0621 05:25:15.986429 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:15.986441 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:15.986517 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:15.986669 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:15.987895 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:15.988913 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:15.989778 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:15.990345 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:15.990473 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:15.993132 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:15.993136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:15.993137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:15.993150 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:15.995139 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:15.996116 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:15.996121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:15.996122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:15.996279 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:15.996607 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:15.996608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:15.996608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:15.996866 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:15.997061 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:15.997159 (+ 98us) server_negotiation.cc:299] Negotiation successful
0621 05:25:15.997280 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
I20260621 05:25:15.998482 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:25:15.999905 23844 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:25:16.000600 23844 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:25:16.006333 23836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:15.999875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38049 (local address 127.2.32.62:34975)
0621 05:25:15.999916 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:15.999918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:16.000002 (+ 84us) server_negotiation.cc:407] Connection header received
0621 05:25:16.000086 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:16.000087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:16.000096 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:16.000152 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:16.000728 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:16.001214 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:16.001918 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:16.002155 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:16.002241 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:16.003518 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:16.003519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:16.003520 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:16.003527 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:16.004971 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:16.005334 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:16.005335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:16.005335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:16.005393 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:16.005622 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:16.005622 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:16.005623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:16.005731 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:16.005839 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:16.005957 (+ 118us) server_negotiation.cc:299] Negotiation successful
0621 05:25:16.006020 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:25:16.013224 23844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b8da8b5e326404588e992abe11ccffb: Generated new TSK 0
I20260621 05:25:16.013391 23844 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:25:16.015945 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 23762
W20260621 05:25:16.020144 23747 connection.cc:535] client connection to 127.2.32.62:34975 recv error: Network error: failed to read from TLS socket (remote: 127.2.32.62:34975): Connection reset by peer (error 104)
2026-06-21T05:25:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:25:16 dist-test-slave-b357 krb5kdc[23861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:25:16 dist-test-slave-b357 krb5kdc[23861](info): set up 2 sockets
Jun 21 05:25:16 dist-test-slave-b357 krb5kdc[23861](info): commencing operation
krb5kdc: starting...
W20260621 05:25:17.075412 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:25:17 dist-test-slave-b357 krb5kdc[23861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:25:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:25:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:25:17.208022 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:43007
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:34755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:43007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:25:17.290508 23874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:17.290755 23874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:17.290791 23874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:17.291982 23874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:17.292050 23874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:17.292078 23874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:17.292111 23874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:17.292135 23874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:17.293558 23874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:34755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:43007
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:43007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:17.293841 23874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:17.294054 23874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:17.298995 23882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:17.299108 23879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:17.298996 23880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:17.299783 23874 server_base.cc:1034] running on GCE node
I20260621 05:25:17.300048 23874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:17.300340 23874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:17.301560 23874 hybrid_clock.cc:648] HybridClock initialized: now 1782019517301531 us; error 47 us; skew 500 ppm
Jun 21 05:25:17 dist-test-slave-b357 krb5kdc[23861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019517, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:17.309201 23874 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:25:17.309679 23874 webserver.cc:460] Webserver started at http://127.2.32.62:40119/ using document root <none> and password file <none>
I20260621 05:25:17.309940 23874 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:17.310009 23874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:17.310168 23874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:17.311295 23874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "b1437643dd494b2fa68dd0e7d2a097ce"
format_stamp: "Formatted at 2026-06-21 05:25:17 on dist-test-slave-b357"
server_key: "334e0706cf3c3e216c1132428975601e"
server_key_iv: "737a48b477ec848c1f479fa048b07f54"
server_key_version: "encryptionkey@0"
I20260621 05:25:17.311653 23874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "b1437643dd494b2fa68dd0e7d2a097ce"
format_stamp: "Formatted at 2026-06-21 05:25:17 on dist-test-slave-b357"
server_key: "334e0706cf3c3e216c1132428975601e"
server_key_iv: "737a48b477ec848c1f479fa048b07f54"
server_key_version: "encryptionkey@0"
I20260621 05:25:17.313107 23874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260621 05:25:17.314324 23889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:17.314591 23874 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:25:17.314726 23874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "b1437643dd494b2fa68dd0e7d2a097ce"
format_stamp: "Formatted at 2026-06-21 05:25:17 on dist-test-slave-b357"
server_key: "334e0706cf3c3e216c1132428975601e"
server_key_iv: "737a48b477ec848c1f479fa048b07f54"
server_key_version: "encryptionkey@0"
I20260621 05:25:17.314813 23874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:17.325732 23874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:17.328323 23874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:17.328500 23874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:17.332623 23874 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:43007
I20260621 05:25:17.332672 23941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:43007 every 8 connection(s)
I20260621 05:25:17.333041 23874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:25:17.333889 23942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:17.333928 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 23874
I20260621 05:25:17.334043 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:25:17.334251 2176 external_mini_cluster.cc:1428] Setting key 19642d2ce516140b463b1868a35f4a34
I20260621 05:25:17.337116 23942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce: Bootstrap starting.
I20260621 05:25:17.337791 23942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:17.338114 23942 log.cc:824] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:17.339264 23942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce: No bootstrap required, opened a new log
I20260621 05:25:17.340512 23945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:17.334728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56365 (local address 127.2.32.62:43007)
0621 05:25:17.334901 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:17.334909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:17.334942 (+ 33us) server_negotiation.cc:407] Connection header received
0621 05:25:17.335444 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:17.335450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:17.335493 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:17.335605 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:17.335903 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:17.336787 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:17.337962 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:17.338689 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:17.338915 (+ 226us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:17.339741 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:17.339745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:17.339745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:17.339947 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:17.339989 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:17.340004 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:17.340100 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:17.340281 (+ 181us) server_negotiation.cc:299] Negotiation successful
0621 05:25:17.340309 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260621 05:25:17.341636 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:25:17.341550 23942 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1437643dd494b2fa68dd0e7d2a097ce" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43007 } }
I20260621 05:25:17.341709 23942 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:25:17.341740 23942 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1437643dd494b2fa68dd0e7d2a097ce, State: Initialized, Role: FOLLOWER
I20260621 05:25:17.341849 23942 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1437643dd494b2fa68dd0e7d2a097ce" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43007 } }
I20260621 05:25:17.341912 23942 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:25:17.341938 23942 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:25:17.341974 23942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:17.342674 23942 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1437643dd494b2fa68dd0e7d2a097ce" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43007 } }
I20260621 05:25:17.342793 23942 leader_election.cc:302] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1437643dd494b2fa68dd0e7d2a097ce; no voters:
I20260621 05:25:17.342948 23942 leader_election.cc:288] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:25:17.343007 23951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:25:17.343190 23951 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce [term 1 LEADER]: Becoming Leader. State: Replica: b1437643dd494b2fa68dd0e7d2a097ce, State: Running, Role: LEADER
I20260621 05:25:17.343226 23942 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:17.343293 23951 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1437643dd494b2fa68dd0e7d2a097ce" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43007 } }
I20260621 05:25:17.343603 23951 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1437643dd494b2fa68dd0e7d2a097ce. Latest consensus state: current_term: 1 leader_uuid: "b1437643dd494b2fa68dd0e7d2a097ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1437643dd494b2fa68dd0e7d2a097ce" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43007 } } }
I20260621 05:25:17.343713 23951 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:17.343616 23952 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1437643dd494b2fa68dd0e7d2a097ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1437643dd494b2fa68dd0e7d2a097ce" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 43007 } } }
I20260621 05:25:17.343919 23952 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:17.343920 23957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:25:17.344444 23957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:25:17.346457 23957 catalog_manager.cc:1347] Generated new cluster ID: 2a1c0db96b474aa5aaa322d8465f567c
I20260621 05:25:17.346534 23957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:25:17.347100 23945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:17.343329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50097 (local address 127.2.32.62:43007)
0621 05:25:17.343362 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:17.343364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:17.343600 (+ 236us) server_negotiation.cc:407] Connection header received
0621 05:25:17.343693 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:17.343694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:17.343705 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:17.343785 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:17.344013 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:17.344786 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:17.345796 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:17.346020 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:17.346053 (+ 33us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:17.346834 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:17.346836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:17.346836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:17.346841 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:17.346855 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:17.346860 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:17.346929 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:17.347000 (+ 71us) server_negotiation.cc:299] Negotiation successful
0621 05:25:17.347011 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260621 05:25:17.363608 23957 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:25:17.364117 23957 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:25:17.367282 23957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1437643dd494b2fa68dd0e7d2a097ce: Generated new TSK 0
I20260621 05:25:17.367455 23957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:25:17.376088 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 23874
W20260621 05:25:17.381459 23859 connection.cc:535] client connection to 127.2.32.62:43007 recv error: Network error: failed to read from TLS socket (remote: 127.2.32.62:43007): Connection reset by peer (error 104)
2026-06-21T05:25:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:25:17 dist-test-slave-b357 krb5kdc[23973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:25:17 dist-test-slave-b357 krb5kdc[23973](info): set up 2 sockets
Jun 21 05:25:17 dist-test-slave-b357 krb5kdc[23973](info): commencing operation
krb5kdc: starting...
W20260621 05:25:18.431589 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:25:18 dist-test-slave-b357 krb5kdc[23973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:25:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:25:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:25:18.561685 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:45125
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:33123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:45125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:25:18.654175 23986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:18.654393 23986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:18.654420 23986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:18.656563 23986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:18.656648 23986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:18.656666 23986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:18.656680 23986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:18.656694 23986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:18.659314 23986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:33123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:45125
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:45125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:18.659543 23986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:18.659710 23986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:18.664618 23992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:18.664640 23994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:18.664767 23986 server_base.cc:1034] running on GCE node
W20260621 05:25:18.664633 23991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:18.665100 23986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:18.665349 23986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:18.666569 23986 hybrid_clock.cc:648] HybridClock initialized: now 1782019518666567 us; error 41 us; skew 500 ppm
Jun 21 05:25:18 dist-test-slave-b357 krb5kdc[23973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019518, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:18.669183 23986 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:25:18.669611 23986 webserver.cc:460] Webserver started at http://127.2.32.62:35897/ using document root <none> and password file <none>
I20260621 05:25:18.669811 23986 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:18.669872 23986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:18.670038 23986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:18.671097 23986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "56cd456c279f444eb760f93a822db788"
format_stamp: "Formatted at 2026-06-21 05:25:18 on dist-test-slave-b357"
server_key: "8fe11d09585f346d71fda9bdac09c3d0"
server_key_iv: "17af2adef382b464d3f656d385e12024"
server_key_version: "encryptionkey@0"
I20260621 05:25:18.671518 23986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "56cd456c279f444eb760f93a822db788"
format_stamp: "Formatted at 2026-06-21 05:25:18 on dist-test-slave-b357"
server_key: "8fe11d09585f346d71fda9bdac09c3d0"
server_key_iv: "17af2adef382b464d3f656d385e12024"
server_key_version: "encryptionkey@0"
I20260621 05:25:18.672960 23986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:25:18.674157 24001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:18.674355 23986 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:25:18.674449 23986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "56cd456c279f444eb760f93a822db788"
format_stamp: "Formatted at 2026-06-21 05:25:18 on dist-test-slave-b357"
server_key: "8fe11d09585f346d71fda9bdac09c3d0"
server_key_iv: "17af2adef382b464d3f656d385e12024"
server_key_version: "encryptionkey@0"
I20260621 05:25:18.674503 23986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:18.677637 23986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:18.679867 23986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:18.680009 23986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:18.683759 23986 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:45125
I20260621 05:25:18.683820 24053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:45125 every 8 connection(s)
I20260621 05:25:18.684111 23986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:25:18.685592 24054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:18.688148 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 23986
I20260621 05:25:18.688258 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:25:18.688477 2176 external_mini_cluster.cc:1428] Setting key a5cb372372751e475bd783978623e9fa
I20260621 05:25:18.688911 24054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788: Bootstrap starting.
I20260621 05:25:18.689796 24054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:18.690173 24054 log.cc:824] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:18.691282 24054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788: No bootstrap required, opened a new log
I20260621 05:25:18.691815 24057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:18.689251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47059 (local address 127.2.32.62:45125)
0621 05:25:18.689503 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:18.689515 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:18.689554 (+ 39us) server_negotiation.cc:407] Connection header received
0621 05:25:18.690313 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:18.690324 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:18.690365 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:18.690552 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:18.691305 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:18.691310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:18.691311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:18.691316 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:18.691398 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:18.691415 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:18.691523 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:18.691597 (+ 74us) server_negotiation.cc:299] Negotiation successful
0621 05:25:18.691642 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":100,"threads_started":1}
I20260621 05:25:18.693136 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:25:18.693992 24054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cd456c279f444eb760f93a822db788" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45125 } }
I20260621 05:25:18.694226 24054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:25:18.694424 24054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56cd456c279f444eb760f93a822db788, State: Initialized, Role: FOLLOWER
I20260621 05:25:18.694653 24054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cd456c279f444eb760f93a822db788" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45125 } }
I20260621 05:25:18.694808 24054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:25:18.694861 24054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:25:18.694923 24054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:18.695741 24057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:18.694633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39055 (local address 127.2.32.62:45125)
0621 05:25:18.694677 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:18.694679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:18.694821 (+ 142us) server_negotiation.cc:407] Connection header received
0621 05:25:18.694922 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:18.694923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:18.694930 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:18.694961 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:18.695560 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:18.695562 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:18.695562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:18.695566 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:18.695579 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:18.695583 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:18.695637 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:18.695689 (+ 52us) server_negotiation.cc:299] Negotiation successful
0621 05:25:18.695700 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:25:18.695979 24054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cd456c279f444eb760f93a822db788" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45125 } }
I20260621 05:25:18.696112 24054 leader_election.cc:302] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56cd456c279f444eb760f93a822db788; no voters:
I20260621 05:25:18.696277 24054 leader_election.cc:288] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:25:18.696295 24064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:25:18.696409 24064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788 [term 1 LEADER]: Becoming Leader. State: Replica: 56cd456c279f444eb760f93a822db788, State: Running, Role: LEADER
I20260621 05:25:18.696516 24064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cd456c279f444eb760f93a822db788" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45125 } }
I20260621 05:25:18.696543 24054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:18.696872 24065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56cd456c279f444eb760f93a822db788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cd456c279f444eb760f93a822db788" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45125 } } }
I20260621 05:25:18.697016 24065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:18.696871 24066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56cd456c279f444eb760f93a822db788. Latest consensus state: current_term: 1 leader_uuid: "56cd456c279f444eb760f93a822db788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cd456c279f444eb760f93a822db788" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45125 } } }
I20260621 05:25:18.697218 24066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:18.697335 24070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:25:18.697865 24070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:25:18.700102 24070 catalog_manager.cc:1347] Generated new cluster ID: e846aae6ca044cf09ed92d40d292a140
I20260621 05:25:18.700161 24070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:25:18.724422 24070 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:25:18.724625 24070 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:25:18.731351 24070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56cd456c279f444eb760f93a822db788: Generated new TSK 0
I20260621 05:25:18.731745 24070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:25:18.739229 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 23986
2026-06-21T05:25:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:25:18 dist-test-slave-b357 krb5kdc[24085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:25:18 dist-test-slave-b357 krb5kdc[24085](info): set up 2 sockets
Jun 21 05:25:18 dist-test-slave-b357 krb5kdc[24085](info): commencing operation
krb5kdc: starting...
W20260621 05:25:19.799322 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:25:19 dist-test-slave-b357 krb5kdc[24085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:25:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:25:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:25:19.927060 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:33823
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:33823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:25:20.013842 24098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:20.014161 24098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:20.014214 24098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:20.015676 24098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:20.015756 24098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:20.015787 24098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:20.015832 24098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:20.015861 24098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:20.017602 24098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:33823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:33823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:20.018070 24098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:20.018419 24098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:20.023516 24104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:20.023695 24106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:20.023527 24103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:20.023940 24098 server_base.cc:1034] running on GCE node
I20260621 05:25:20.024334 24098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:20.024830 24098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:20.026072 24098 hybrid_clock.cc:648] HybridClock initialized: now 1782019520026027 us; error 62 us; skew 500 ppm
Jun 21 05:25:20 dist-test-slave-b357 krb5kdc[24085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019520, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:20.028942 24098 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:25:20.029376 24098 webserver.cc:460] Webserver started at http://127.2.32.62:35965/ using document root <none> and password file <none>
I20260621 05:25:20.029562 24098 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:20.029606 24098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:20.029690 24098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:20.030704 24098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "af1ad3f761214097bb809ebe396fae9f"
format_stamp: "Formatted at 2026-06-21 05:25:20 on dist-test-slave-b357"
server_key: "62c79ebdd881805130524a32640882ec"
server_key_iv: "d412ee3694f773b5425b134b1fbc72ee"
server_key_version: "encryptionkey@0"
I20260621 05:25:20.031003 24098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "af1ad3f761214097bb809ebe396fae9f"
format_stamp: "Formatted at 2026-06-21 05:25:20 on dist-test-slave-b357"
server_key: "62c79ebdd881805130524a32640882ec"
server_key_iv: "d412ee3694f773b5425b134b1fbc72ee"
server_key_version: "encryptionkey@0"
I20260621 05:25:20.032537 24098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:25:20.033644 24113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:20.033943 24098 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:25:20.034070 24098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "af1ad3f761214097bb809ebe396fae9f"
format_stamp: "Formatted at 2026-06-21 05:25:20 on dist-test-slave-b357"
server_key: "62c79ebdd881805130524a32640882ec"
server_key_iv: "d412ee3694f773b5425b134b1fbc72ee"
server_key_version: "encryptionkey@0"
I20260621 05:25:20.034161 24098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:20.059002 24098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:20.061453 24098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:20.061606 24098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:20.066488 24098 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:33823
I20260621 05:25:20.066876 24098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:25:20.066884 24165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:33823 every 8 connection(s)
I20260621 05:25:20.067703 24166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:20.070134 24166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f: Bootstrap starting.
I20260621 05:25:20.070724 24166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:20.071008 24166 log.cc:824] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:20.071712 24166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f: No bootstrap required, opened a new log
I20260621 05:25:20.072981 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 24098
I20260621 05:25:20.073114 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:25:20.073278 2176 external_mini_cluster.cc:1428] Setting key 48edb497f2abaa7b1a7860184e22a8c6
I20260621 05:25:20.073635 24166 raft_consensus.cc:348] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1ad3f761214097bb809ebe396fae9f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 33823 } }
I20260621 05:25:20.073861 24166 raft_consensus.cc:374] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:25:20.073925 24166 raft_consensus.cc:729] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af1ad3f761214097bb809ebe396fae9f, State: Initialized, Role: FOLLOWER
I20260621 05:25:20.074097 24166 consensus_queue.cc:260] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1ad3f761214097bb809ebe396fae9f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 33823 } }
I20260621 05:25:20.074167 24166 raft_consensus.cc:388] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:25:20.074189 24166 raft_consensus.cc:482] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:25:20.074252 24166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:20.075009 24166 raft_consensus.cc:504] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1ad3f761214097bb809ebe396fae9f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 33823 } }
I20260621 05:25:20.075147 24166 leader_election.cc:302] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af1ad3f761214097bb809ebe396fae9f; no voters:
I20260621 05:25:20.075311 24166 leader_election.cc:288] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:25:20.075387 24171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:25:20.075603 24171 raft_consensus.cc:686] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f [term 1 LEADER]: Becoming Leader. State: Replica: af1ad3f761214097bb809ebe396fae9f, State: Running, Role: LEADER
I20260621 05:25:20.075742 24171 consensus_queue.cc:237] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1ad3f761214097bb809ebe396fae9f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 33823 } }
I20260621 05:25:20.075891 24166 sys_catalog.cc:564] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:20.076203 24172 sys_catalog.cc:455] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af1ad3f761214097bb809ebe396fae9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1ad3f761214097bb809ebe396fae9f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 33823 } } }
I20260621 05:25:20.076278 24172 sys_catalog.cc:458] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:20.076478 24173 sys_catalog.cc:455] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader af1ad3f761214097bb809ebe396fae9f. Latest consensus state: current_term: 1 leader_uuid: "af1ad3f761214097bb809ebe396fae9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1ad3f761214097bb809ebe396fae9f" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 33823 } } }
I20260621 05:25:20.076835 24173 sys_catalog.cc:458] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:20.076759 24175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:25:20.077423 24175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:25:20.079223 24175 catalog_manager.cc:1347] Generated new cluster ID: 55c6bc53c0804d5bb181b37fa0e6cd3e
I20260621 05:25:20.079281 24175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 21 05:25:20 dist-test-slave-b357 krb5kdc[24085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:25:20.084816 24170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:20.073836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40861 (local address 127.2.32.62:33823)
0621 05:25:20.074056 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:20.074064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:20.074108 (+ 44us) server_negotiation.cc:407] Connection header received
0621 05:25:20.074244 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:20.074252 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:20.074306 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:20.074432 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:20.075276 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:20.077122 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:20.078297 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:20.078947 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:20.079055 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:20.081569 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:20.081573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:20.081573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:20.081585 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:20.083308 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:20.083674 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:20.083676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:20.083677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:20.083776 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:20.083987 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:20.083988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:20.083988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:20.084227 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:20.084401 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:20.084469 (+ 68us) server_negotiation.cc:299] Negotiation successful
0621 05:25:20.084606 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":101,"threads_started":1}
I20260621 05:25:20.085678 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:25:20.093997 24170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:20.087038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36677 (local address 127.2.32.62:33823)
0621 05:25:20.087075 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:20.087077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:20.087199 (+ 122us) server_negotiation.cc:407] Connection header received
0621 05:25:20.087271 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:20.087271 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:20.087284 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:20.087338 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:20.087929 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:20.089146 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:20.089887 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:20.090109 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:20.090191 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:20.091429 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:20.091431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:20.091431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:20.091438 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:20.092854 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:20.093214 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:20.093215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:20.093215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:20.093269 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:20.093476 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:20.093477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:20.093478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:20.093593 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:20.093712 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:20.093818 (+ 106us) server_negotiation.cc:299] Negotiation successful
0621 05:25:20.093892 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:25:20.114898 24175 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:25:20.115522 24175 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:25:20.122846 24175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af1ad3f761214097bb809ebe396fae9f: Generated new TSK 0
I20260621 05:25:20.123025 24175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:25:20.125233 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 24098
2026-06-21T05:25:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:25:20 dist-test-slave-b357 krb5kdc[24197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:25:20 dist-test-slave-b357 krb5kdc[24197](info): set up 2 sockets
Jun 21 05:25:20 dist-test-slave-b357 krb5kdc[24197](info): commencing operation
krb5kdc: starting...
W20260621 05:25:21.179304 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:25:21 dist-test-slave-b357 krb5kdc[24197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:25:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:25:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:25:21.306488 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:41185
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:44423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:41185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:25:21.388556 24210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:21.388805 24210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:21.388840 24210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:21.390007 24210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:21.390084 24210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:21.390110 24210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:21.390129 24210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:21.390154 24210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:21.391673 24210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:44423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:41185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:41185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:21.391943 24210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:21.392166 24210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:21.397012 24218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:21.397047 24216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:21.397022 24215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:21.397351 24210 server_base.cc:1034] running on GCE node
I20260621 05:25:21.397661 24210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:21.397939 24210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:21.399303 24210 hybrid_clock.cc:648] HybridClock initialized: now 1782019521399261 us; error 72 us; skew 500 ppm
Jun 21 05:25:21 dist-test-slave-b357 krb5kdc[24197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019521, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:21.402493 24210 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:25:21.402979 24210 webserver.cc:460] Webserver started at http://127.2.32.62:38487/ using document root <none> and password file <none>
I20260621 05:25:21.403268 24210 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:21.403338 24210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:21.403487 24210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:21.404536 24210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "8c344dea818d4da386b440c36325fa00"
format_stamp: "Formatted at 2026-06-21 05:25:21 on dist-test-slave-b357"
server_key: "471233fcf149cb4c214dd95abb7a5a5b"
server_key_iv: "3e9d45d02684395732bbcf699c1f54fa"
server_key_version: "encryptionkey@0"
I20260621 05:25:21.404896 24210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "8c344dea818d4da386b440c36325fa00"
format_stamp: "Formatted at 2026-06-21 05:25:21 on dist-test-slave-b357"
server_key: "471233fcf149cb4c214dd95abb7a5a5b"
server_key_iv: "3e9d45d02684395732bbcf699c1f54fa"
server_key_version: "encryptionkey@0"
I20260621 05:25:21.406279 24210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260621 05:25:21.407260 24225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:21.407544 24210 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:25:21.407645 24210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "8c344dea818d4da386b440c36325fa00"
format_stamp: "Formatted at 2026-06-21 05:25:21 on dist-test-slave-b357"
server_key: "471233fcf149cb4c214dd95abb7a5a5b"
server_key_iv: "3e9d45d02684395732bbcf699c1f54fa"
server_key_version: "encryptionkey@0"
I20260621 05:25:21.407737 24210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:21.424291 24210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:21.426762 24210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:21.426940 24210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:21.431005 24210 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:41185
I20260621 05:25:21.431049 24277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:41185 every 8 connection(s)
I20260621 05:25:21.431432 24210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:25:21.432150 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 24210
I20260621 05:25:21.432260 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:25:21.432421 24278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:21.432516 2176 external_mini_cluster.cc:1428] Setting key 6d3819d6db63e1660b67f37091507071
I20260621 05:25:21.437008 24278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00: Bootstrap starting.
I20260621 05:25:21.437824 24278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:21.438139 24278 log.cc:824] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:21.439215 24278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00: No bootstrap required, opened a new log
I20260621 05:25:21.439543 24281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:21.433035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56325 (local address 127.2.32.62:41185)
0621 05:25:21.433231 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:21.433240 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:21.433274 (+ 34us) server_negotiation.cc:407] Connection header received
0621 05:25:21.433738 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:21.433746 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:21.433817 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:21.433965 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:21.434170 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:21.436135 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:21.436966 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:21.437691 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:21.437818 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:21.438526 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:21.438531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:21.438531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:21.438751 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:21.438786 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:21.438797 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:21.438853 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:21.439049 (+ 196us) server_negotiation.cc:299] Negotiation successful
0621 05:25:21.439125 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260621 05:25:21.440616 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:25:21.441529 24278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c344dea818d4da386b440c36325fa00" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 41185 } }
I20260621 05:25:21.441686 24278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:25:21.441733 24278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c344dea818d4da386b440c36325fa00, State: Initialized, Role: FOLLOWER
I20260621 05:25:21.441852 24278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c344dea818d4da386b440c36325fa00" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 41185 } }
I20260621 05:25:21.441936 24278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:25:21.441991 24278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:25:21.442042 24278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:21.442611 24278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c344dea818d4da386b440c36325fa00" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 41185 } }
I20260621 05:25:21.442740 24278 leader_election.cc:302] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c344dea818d4da386b440c36325fa00; no voters:
I20260621 05:25:21.442917 24278 leader_election.cc:288] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:25:21.442960 24288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:25:21.443099 24288 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00 [term 1 LEADER]: Becoming Leader. State: Replica: 8c344dea818d4da386b440c36325fa00, State: Running, Role: LEADER
I20260621 05:25:21.443259 24288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c344dea818d4da386b440c36325fa00" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 41185 } }
I20260621 05:25:21.443260 24278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:21.443720 24289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c344dea818d4da386b440c36325fa00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c344dea818d4da386b440c36325fa00" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 41185 } } }
I20260621 05:25:21.443719 24290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c344dea818d4da386b440c36325fa00. Latest consensus state: current_term: 1 leader_uuid: "8c344dea818d4da386b440c36325fa00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c344dea818d4da386b440c36325fa00" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 41185 } } }
I20260621 05:25:21.443818 24289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:21.443822 24290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:21.444115 24294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:25:21.444855 24294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:25:21.445202 24281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:21.441936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50007 (local address 127.2.32.62:41185)
0621 05:25:21.441975 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:21.441977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:21.442051 (+ 74us) server_negotiation.cc:407] Connection header received
0621 05:25:21.442120 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:21.442121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:21.442133 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:21.442199 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:21.442340 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:21.442870 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:21.443826 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:21.444133 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:21.444182 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:21.444926 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:21.444927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:21.444928 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:21.444933 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:21.444948 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:21.444952 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:21.444981 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:21.445099 (+ 118us) server_negotiation.cc:299] Negotiation successful
0621 05:25:21.445120 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:25:21.446511 24294 catalog_manager.cc:1347] Generated new cluster ID: f6747ae6688849c28bce761049bd70ef
I20260621 05:25:21.446563 24294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:25:21.472421 24294 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:25:21.473482 24294 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:25:21.484540 24294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c344dea818d4da386b440c36325fa00: Generated new TSK 0
I20260621 05:25:21.484738 24294 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:25:21.487746 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 24210
2026-06-21T05:25:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:25:21 dist-test-slave-b357 krb5kdc[24309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:25:21 dist-test-slave-b357 krb5kdc[24309](info): set up 2 sockets
Jun 21 05:25:21 dist-test-slave-b357 krb5kdc[24309](info): commencing operation
krb5kdc: starting...
W20260621 05:25:22.555729 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:25:22 dist-test-slave-b357 krb5kdc[24309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:25:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:25:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:25:22.688247 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:37391
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:38829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:37391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:25:22.768522 24322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:22.768713 24322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:22.768735 24322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:22.769767 24322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:22.769809 24322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:22.769824 24322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:22.769836 24322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:22.769848 24322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:22.771260 24322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:38829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:37391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:37391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:22.771442 24322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:22.771600 24322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:22.777275 24327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:22.777383 24330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:22.777396 24328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:22.777710 24322 server_base.cc:1034] running on GCE node
I20260621 05:25:22.778045 24322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:22.778349 24322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:22.779563 24322 hybrid_clock.cc:648] HybridClock initialized: now 1782019522779551 us; error 31 us; skew 500 ppm
Jun 21 05:25:22 dist-test-slave-b357 krb5kdc[24309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019522, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:22.785815 24322 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:25:22.786515 24322 webserver.cc:460] Webserver started at http://127.2.32.62:37855/ using document root <none> and password file <none>
I20260621 05:25:22.786811 24322 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:22.786890 24322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:22.787055 24322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:22.788362 24322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "04a66a6b2c69404386d57de74f3cb4d6"
format_stamp: "Formatted at 2026-06-21 05:25:22 on dist-test-slave-b357"
server_key: "b4837a38b92d33bb91c201a7e075ddc3"
server_key_iv: "55d384e75c5eab1bd2a3a998437cff6e"
server_key_version: "encryptionkey@0"
I20260621 05:25:22.788774 24322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "04a66a6b2c69404386d57de74f3cb4d6"
format_stamp: "Formatted at 2026-06-21 05:25:22 on dist-test-slave-b357"
server_key: "b4837a38b92d33bb91c201a7e075ddc3"
server_key_iv: "55d384e75c5eab1bd2a3a998437cff6e"
server_key_version: "encryptionkey@0"
I20260621 05:25:22.790343 24322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 05:25:22.791409 24337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:22.791757 24322 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 05:25:22.791904 24322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "04a66a6b2c69404386d57de74f3cb4d6"
format_stamp: "Formatted at 2026-06-21 05:25:22 on dist-test-slave-b357"
server_key: "b4837a38b92d33bb91c201a7e075ddc3"
server_key_iv: "55d384e75c5eab1bd2a3a998437cff6e"
server_key_version: "encryptionkey@0"
I20260621 05:25:22.791994 24322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:22.795928 24322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:22.798203 24322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:22.798363 24322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:22.802420 24322 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:37391
I20260621 05:25:22.802451 24389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:37391 every 8 connection(s)
I20260621 05:25:22.802788 24322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:25:22.803032 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 24322
I20260621 05:25:22.803181 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:25:22.803380 2176 external_mini_cluster.cc:1428] Setting key 9ea9501293071991bbe82b8dca5ff7e9
I20260621 05:25:22.803933 24390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:22.805787 24393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:22.803910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55305 (local address 127.2.32.62:37391)
0621 05:25:22.804154 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:22.804161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:22.804189 (+ 28us) server_negotiation.cc:407] Connection header received
0621 05:25:22.804622 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:22.804627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:22.804650 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:22.804785 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:22.805389 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:22.805392 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:22.805393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:22.805397 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:22.805442 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:22.805453 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:22.805528 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:22.805595 (+ 67us) server_negotiation.cc:299] Negotiation successful
0621 05:25:22.805634 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":99,"threads_started":1}
I20260621 05:25:22.806846 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:25:22.807181 24390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6: Bootstrap starting.
I20260621 05:25:22.807835 24390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:22.808158 24390 log.cc:824] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:22.808977 24390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6: No bootstrap required, opened a new log
I20260621 05:25:22.809546 24393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:22.808191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56089 (local address 127.2.32.62:37391)
0621 05:25:22.808225 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:22.808227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:22.808496 (+ 269us) server_negotiation.cc:407] Connection header received
0621 05:25:22.808544 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:22.808545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:22.808550 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:22.808605 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:22.809149 (+ 544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:22.809150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:22.809150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 05:25:22.809155 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:22.809168 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 05:25:22.809173 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:22.809231 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:22.809309 (+ 78us) server_negotiation.cc:299] Negotiation successful
0621 05:25:22.809319 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:25:22.810937 24390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a66a6b2c69404386d57de74f3cb4d6" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 37391 } }
I20260621 05:25:22.811105 24390 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 05:25:22.811143 24390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a66a6b2c69404386d57de74f3cb4d6, State: Initialized, Role: FOLLOWER
I20260621 05:25:22.811348 24390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a66a6b2c69404386d57de74f3cb4d6" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 37391 } }
I20260621 05:25:22.811453 24390 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 05:25:22.811497 24390 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 05:25:22.811549 24390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:22.812044 24390 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a66a6b2c69404386d57de74f3cb4d6" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 37391 } }
I20260621 05:25:22.812161 24390 leader_election.cc:302] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04a66a6b2c69404386d57de74f3cb4d6; no voters:
I20260621 05:25:22.812325 24390 leader_election.cc:288] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 05:25:22.812435 24400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:25:22.812608 24390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:22.812671 24400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6 [term 1 LEADER]: Becoming Leader. State: Replica: 04a66a6b2c69404386d57de74f3cb4d6, State: Running, Role: LEADER
I20260621 05:25:22.812934 24400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a66a6b2c69404386d57de74f3cb4d6" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 37391 } }
I20260621 05:25:22.813495 24402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04a66a6b2c69404386d57de74f3cb4d6. Latest consensus state: current_term: 1 leader_uuid: "04a66a6b2c69404386d57de74f3cb4d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a66a6b2c69404386d57de74f3cb4d6" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 37391 } } }
I20260621 05:25:22.813632 24402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:22.813472 24401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04a66a6b2c69404386d57de74f3cb4d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a66a6b2c69404386d57de74f3cb4d6" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 37391 } } }
I20260621 05:25:22.813876 24401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:22.813928 24409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:25:22.814457 24409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:25:22.816352 24409 catalog_manager.cc:1347] Generated new cluster ID: 2d77fc0030074debba1db036023dbd2c
I20260621 05:25:22.816414 24409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:25:22.827353 24409 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:25:22.827559 24409 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:25:22.841104 24409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04a66a6b2c69404386d57de74f3cb4d6: Generated new TSK 0
I20260621 05:25:22.841306 24409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 05:25:22.845021 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 24322
2026-06-21T05:25:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1356 ms)
[----------] 18 tests from AuthTokenIssuingTest (24580 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:25:22 dist-test-slave-b357 krb5kdc[24421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:25:22 dist-test-slave-b357 krb5kdc[24421](info): set up 2 sockets
Jun 21 05:25:22 dist-test-slave-b357 krb5kdc[24421](info): commencing operation
krb5kdc: starting...
W20260621 05:25:23.915433 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:25:23 dist-test-slave-b357 krb5kdc[24421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:25:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:25:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:25:24.043102 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:45803
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:45803,127.2.32.61:40249,127.2.32.60:33391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:25:24.123265 24434 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 05:25:24.123459 24434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:24.123476 24434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:24.123497 24434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:24.124606 24434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:24.124655 24434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:24.124668 24434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:24.124681 24434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:24.124693 24434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:24.126156 24434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:45803,127.2.32.61:40249,127.2.32.60:33391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:45803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:24.126412 24434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:24.126664 24434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:24.131973 24439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:24.132040 24440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:24.132051 24434 server_base.cc:1034] running on GCE node
W20260621 05:25:24.132059 24442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:24.132576 24434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:24.132957 24434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:24.134151 24434 hybrid_clock.cc:648] HybridClock initialized: now 1782019524134128 us; error 54 us; skew 500 ppm
Jun 21 05:25:24 dist-test-slave-b357 krb5kdc[24421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019524, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:24.137145 24434 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:25:24.137758 24434 webserver.cc:460] Webserver started at http://127.2.32.62:35481/ using document root <none> and password file <none>
I20260621 05:25:24.138016 24434 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:24.138087 24434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:24.138234 24434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:24.139353 24434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "5d635affcecb468fa7ada5aa32e7856c"
format_stamp: "Formatted at 2026-06-21 05:25:24 on dist-test-slave-b357"
server_key: "cf2e33aaef66fe6284cb5e89d9f9b12f"
server_key_iv: "b57321d7db7b300cde6b2ce3e43dd8a7"
server_key_version: "encryptionkey@0"
I20260621 05:25:24.139711 24434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "5d635affcecb468fa7ada5aa32e7856c"
format_stamp: "Formatted at 2026-06-21 05:25:24 on dist-test-slave-b357"
server_key: "cf2e33aaef66fe6284cb5e89d9f9b12f"
server_key_iv: "b57321d7db7b300cde6b2ce3e43dd8a7"
server_key_version: "encryptionkey@0"
I20260621 05:25:24.141094 24434 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:25:24.141974 24449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:24.142266 24434 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:25:24.142359 24434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "5d635affcecb468fa7ada5aa32e7856c"
format_stamp: "Formatted at 2026-06-21 05:25:24 on dist-test-slave-b357"
server_key: "cf2e33aaef66fe6284cb5e89d9f9b12f"
server_key_iv: "b57321d7db7b300cde6b2ce3e43dd8a7"
server_key_version: "encryptionkey@0"
I20260621 05:25:24.142416 24434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:24.157631 24434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:24.160171 24434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:24.160287 24434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:24.164060 24434 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:45803
I20260621 05:25:24.164098 24501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:45803 every 8 connection(s)
I20260621 05:25:24.164431 24434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:25:24.165839 24502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:24.168202 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 24434
I20260621 05:25:24.168318 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:25:24.168109 24502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:25:24.168927 2176 external_mini_cluster.cc:1428] Setting key e5041980c54cd448aee174a3f3d39b05
Jun 21 05:25:24 dist-test-slave-b357 krb5kdc[24421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019524, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.32.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.61@KRBTEST.COM" created.
I20260621 05:25:24.181900 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.169019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.2.32.62:45803)
0621 05:25:24.170910 (+ 1891us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.170916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.170936 (+ 20us) server_negotiation.cc:407] Connection header received
0621 05:25:24.171041 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.171044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.171071 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.171214 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.172255 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.173664 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.174736 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.175364 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.175511 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.177732 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:24.177740 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:24.177740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:24.177750 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:24.179819 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.180309 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.180312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.180312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.180467 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.180760 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.180761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.180761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.181031 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:24.181158 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:24.181523 (+ 365us) server_negotiation.cc:299] Negotiation successful
0621 05:25:24.181749 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":1850,"thread_start_us":172,"threads_started":1}
I20260621 05:25:24.181929 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.169012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45803 (local address 127.0.0.1:38150)
0621 05:25:24.169567 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.169623 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0621 05:25:24.170621 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:25:24.171439 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:25:24.171443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:25:24.171884 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:25:24.172136 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:24.172143 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.173795 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.173797 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:24.174633 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:24.174636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.175594 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.175596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:24.175696 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.176269 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:25:24.176277 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:25:24.177630 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:25:24.179930 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:24.179937 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:24.179939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:24.180142 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:24.180582 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:24.180582 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:24.180583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:24.180675 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:24.181197 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:25:24.181204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:25:24.181262 (+ 58us) client_negotiation.cc:769] Sending connection context
0621 05:25:24.181477 (+ 215us) client_negotiation.cc:240] Negotiation successful
0621 05:25:24.181637 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":169,"threads_started":1}
I20260621 05:25:24.182673 24502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:25:24.183059 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.182925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:40249 (local address 127.0.0.1:59246)
0621 05:25:24.182974 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.183021 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.61:40249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260621 05:25:24.183259 24450 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.32.61:40249: connect: Connection refused (error 111)
I20260621 05:25:24.183746 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.183628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:40249 (local address 127.0.0.1:59258)
0621 05:25:24.183705 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.183730 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.61:40249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260621 05:25:24.183967 24502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.61:40249: Network error: Client connection negotiation failed: client connection to 127.2.32.61:40249: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.61.keytab.
Entry for principal kudu/127.2.32.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.61.keytab.
I20260621 05:25:24.198410 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.61:40249
--webserver_interface=127.2.32.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.61.keytab
--principal=kudu/127.2.32.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:45803,127.2.32.61:40249,127.2.32.60:33391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:25:24.233294 24502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } attempt: 1
I20260621 05:25:24.234035 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.233799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:40249 (local address 127.0.0.1:59262)
0621 05:25:24.233880 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.233998 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.61:40249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260621 05:25:24.234611 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.234366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:40249 (local address 127.0.0.1:59266)
0621 05:25:24.234464 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.234543 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.61:40249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260621 05:25:24.234861 24502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.61:40249: Network error: Client connection negotiation failed: client connection to 127.2.32.61:40249: connect: Connection refused (error 111)
W20260621 05:25:24.281831 24509 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 05:25:24.282048 24509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:24.282066 24509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:24.282086 24509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:24.283232 24509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:24.283274 24509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:24.283288 24509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:24.283301 24509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:24.283313 24509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:24.284770 24509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:45803,127.2.32.61:40249,127.2.32.60:33391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.61.keytab
--principal=kudu/127.2.32.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.61:40249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.32.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:24.285009 24509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:24.285257 24509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:24.290305 24514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:24.290305 24517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:24.290367 24515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:24.290493 24509 server_base.cc:1034] running on GCE node
I20260621 05:25:24.290738 24509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:24.291008 24509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:24.292254 24509 hybrid_clock.cc:648] HybridClock initialized: now 1782019524292175 us; error 33 us; skew 500 ppm
Jun 21 05:25:24 dist-test-slave-b357 krb5kdc[24421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019524, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:24.295368 24509 init.cc:377] Logged in from keytab as kudu/127.2.32.61@KRBTEST.COM (short username kudu)
I20260621 05:25:24.295933 24509 webserver.cc:460] Webserver started at http://127.2.32.61:36871/ using document root <none> and password file <none>
I20260621 05:25:24.296216 24509 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:24.296267 24509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:24.296466 24509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:24.297730 24509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-1/data/instance:
uuid: "a5ab9c20160842a1b50d89e2aa226773"
format_stamp: "Formatted at 2026-06-21 05:25:24 on dist-test-slave-b357"
server_key: "2048cdc3568f6d0fa4ebc2f4265db015"
server_key_iv: "7f4b27ab86f689a45b22377b3e2ab043"
server_key_version: "encryptionkey@0"
I20260621 05:25:24.298141 24509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-1/wal/instance:
uuid: "a5ab9c20160842a1b50d89e2aa226773"
format_stamp: "Formatted at 2026-06-21 05:25:24 on dist-test-slave-b357"
server_key: "2048cdc3568f6d0fa4ebc2f4265db015"
server_key_iv: "7f4b27ab86f689a45b22377b3e2ab043"
server_key_version: "encryptionkey@0"
I20260621 05:25:24.299979 24509 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260621 05:25:24.300922 24524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:24.301149 24509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:25:24.301229 24509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-1/wal
uuid: "a5ab9c20160842a1b50d89e2aa226773"
format_stamp: "Formatted at 2026-06-21 05:25:24 on dist-test-slave-b357"
server_key: "2048cdc3568f6d0fa4ebc2f4265db015"
server_key_iv: "7f4b27ab86f689a45b22377b3e2ab043"
server_key_version: "encryptionkey@0"
I20260621 05:25:24.301296 24509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:24.303062 24502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } attempt: 2
I20260621 05:25:24.303542 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.303406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:40249 (local address 127.0.0.1:59282)
0621 05:25:24.303470 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.303502 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.61:40249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 05:25:24.303933 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.303844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:40249 (local address 127.0.0.1:59294)
0621 05:25:24.303893 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.303917 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.61:40249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260621 05:25:24.304052 24502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.61:40249: Network error: Client connection negotiation failed: client connection to 127.2.32.61:40249: connect: Connection refused (error 111)
I20260621 05:25:24.313934 24509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:24.316468 24509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:24.316589 24509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:24.320613 24509 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.61:40249
I20260621 05:25:24.320653 24576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.61:40249 every 8 connection(s)
I20260621 05:25:24.320991 24509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-1/data/info.pb
I20260621 05:25:24.322144 24577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:24.323910 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 24509
I20260621 05:25:24.324031 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-1/wal/instance
I20260621 05:25:24.324234 2176 external_mini_cluster.cc:1428] Setting key 0a62e7e97ca547258ec1e8de0c779a3f
I20260621 05:25:24.324689 24577 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.32.60@KRBTEST.COM; defaulting to no policy
Jun 21 05:25:24 dist-test-slave-b357 krb5kdc[24421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019524, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.61@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.60@KRBTEST.COM" created.
I20260621 05:25:24.335810 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.325621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.2.32.62:45803)
0621 05:25:24.325696 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.325699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.326000 (+ 301us) server_negotiation.cc:407] Connection header received
0621 05:25:24.326570 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.326571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.326586 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.326650 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.327332 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.327890 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.328990 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.329188 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.329295 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.331529 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:24.331531 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:24.331531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:24.331538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:24.333473 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.334262 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.334263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.334263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.334380 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.334802 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.334803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.334804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.334987 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:24.335073 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:24.335544 (+ 471us) server_negotiation.cc:299] Negotiation successful
0621 05:25:24.335665 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 05:25:24.335863 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.325572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45803 (local address 127.0.0.1:38166)
0621 05:25:24.325858 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.325904 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 05:25:24.326460 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:25:24.326716 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:25:24.326719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:25:24.326980 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:25:24.327244 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:24.327248 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.328004 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.328005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:24.328876 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:24.328879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.329314 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.329315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:24.329451 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.329979 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:25:24.329987 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:25:24.331428 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:25:24.333680 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:24.333700 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:24.333703 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:24.334124 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:24.334498 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:24.334499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:24.334499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:24.334626 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:24.335102 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:25:24.335109 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:25:24.335338 (+ 229us) client_negotiation.cc:769] Sending connection context
0621 05:25:24.335459 (+ 121us) client_negotiation.cc:240] Negotiation successful
0621 05:25:24.335623 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":97,"threads_started":1}
I20260621 05:25:24.336987 24577 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:25:24 dist-test-slave-b357 krb5kdc[24421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019524, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.61@KRBTEST.COM for kudu/127.2.32.61@KRBTEST.COM
I20260621 05:25:24.344864 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.337316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:40249 (local address 127.0.0.1:59304)
0621 05:25:24.337380 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.337390 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:25:24.337440 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:25:24.337683 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:25:24.337684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:25:24.337826 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:25:24.337871 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:24.337872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.338728 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.338729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:24.339301 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:24.339303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.339784 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.339784 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:24.339848 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.340289 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:25:24.340294 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:25:24.341865 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:25:24.343773 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:24.343774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:24.343774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:24.343940 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:24.344153 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:24.344154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:24.344154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:24.344209 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:24.344501 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:25:24.344502 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:25:24.344541 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 05:25:24.344607 (+ 66us) client_negotiation.cc:240] Negotiation successful
0621 05:25:24.344781 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":85}
I20260621 05:25:24.344877 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.337388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59304 (local address 127.2.32.61:40249)
0621 05:25:24.337559 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.337562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.337576 (+ 14us) server_negotiation.cc:407] Connection header received
0621 05:25:24.337591 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.337592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.337602 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.337683 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.337967 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.338644 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.339393 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.339698 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.339785 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.342002 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:24.342009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:24.342009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:24.342016 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:24.343667 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.344023 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.344026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.344027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.344085 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.344280 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.344281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.344281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.344426 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:24.344521 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:24.344633 (+ 112us) server_negotiation.cc:299] Negotiation successful
0621 05:25:24.344731 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":106,"threads_started":1}
I20260621 05:25:24.345989 24577 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:25:24.346549 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.346411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:33391 (local address 127.0.0.1:47736)
0621 05:25:24.346475 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.346527 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260621 05:25:24.346814 24528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.32.60:33391: connect: Connection refused (error 111)
I20260621 05:25:24.347498 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.347366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:33391 (local address 127.0.0.1:47742)
0621 05:25:24.347423 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.347460 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260621 05:25:24.347697 24577 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.60:33391: Network error: Client connection negotiation failed: client connection to 127.2.32.60:33391: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.60.keytab.
Entry for principal kudu/127.2.32.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.60.keytab.
I20260621 05:25:24.350493 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.60:33391
--webserver_interface=127.2.32.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.60.keytab
--principal=kudu/127.2.32.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:45803,127.2.32.61:40249,127.2.32.60:33391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:25:24.395186 24502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } attempt: 3
I20260621 05:25:24.396926 24577 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } attempt: 1
I20260621 05:25:24.397308 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.397170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:33391 (local address 127.0.0.1:47744)
0621 05:25:24.397232 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.397271 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260621 05:25:24.397974 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.397869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:33391 (local address 127.0.0.1:47760)
0621 05:25:24.397921 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.397950 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260621 05:25:24.398145 24577 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.60:33391: Network error: Client connection negotiation failed: client connection to 127.2.32.60:33391: connect: Connection refused (error 111)
Jun 21 05:25:24 dist-test-slave-b357 krb5kdc[24421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019524, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for kudu/127.2.32.61@KRBTEST.COM
I20260621 05:25:24.402779 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.395711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59318 (local address 127.2.32.61:40249)
0621 05:25:24.395778 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.395781 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.395793 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:25:24.395867 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.395868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.395881 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.395956 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.396278 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.396781 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.397712 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.398031 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.398099 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.400219 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:24.400220 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:24.400221 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:24.400227 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:24.401626 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.402000 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.402002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.402002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.402070 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.402269 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.402270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.402270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.402387 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:24.402463 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:24.402619 (+ 156us) server_negotiation.cc:299] Negotiation successful
0621 05:25:24.402693 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 05:25:24.402784 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.395646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:40249 (local address 127.0.0.1:59318)
0621 05:25:24.395703 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.395722 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 05:25:24.395795 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:25:24.395962 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:25:24.395963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:25:24.396131 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:25:24.396194 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:24.396196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.396900 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.396901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:24.397625 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:24.397627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.398095 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.398096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:24.398177 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.398688 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:25:24.398693 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:25:24.400106 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:25:24.401741 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:24.401742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:24.401743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:24.401912 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:24.402138 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:24.402139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:24.402140 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:24.402204 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:24.402455 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:25:24.402455 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:25:24.402502 (+ 47us) client_negotiation.cc:769] Sending connection context
0621 05:25:24.402596 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 05:25:24.402693 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260621 05:25:24.403328 24502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:25:24.403630 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.403552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:33391 (local address 127.0.0.1:47764)
0621 05:25:24.403589 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.403614 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260621 05:25:24.403955 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.403867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:33391 (local address 127.0.0.1:47776)
0621 05:25:24.403912 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.403935 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260621 05:25:24.404074 24502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.60:33391: Network error: Client connection negotiation failed: client connection to 127.2.32.60:33391: connect: Connection refused (error 111)
W20260621 05:25:24.434404 24584 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 05:25:24.434615 24584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:24.434631 24584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:24.434653 24584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260621 05:25:24.435254 24502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } attempt: 1
I20260621 05:25:24.435735 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.435609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:33391 (local address 127.0.0.1:47790)
0621 05:25:24.435670 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.435708 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260621 05:25:24.435904 24584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:24.435966 24584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:24.436033 24584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260621 05:25:24.436066 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.435966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:33391 (local address 127.0.0.1:47802)
0621 05:25:24.436016 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.436040 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260621 05:25:24.436084 24584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:24.436106 24584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 05:25:24.436228 24502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.60:33391: Network error: Client connection negotiation failed: client connection to 127.2.32.60:33391: connect: Connection refused (error 111)
I20260621 05:25:24.437724 24584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:45803,127.2.32.61:40249,127.2.32.60:33391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.60.keytab
--principal=kudu/127.2.32.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.60:33391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.32.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:24.437990 24584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:24.438212 24584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:24.442781 24589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:24.442799 24592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:24.442798 24590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:24.443148 24584 server_base.cc:1034] running on GCE node
I20260621 05:25:24.443387 24584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:24.443670 24584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:24.444875 24584 hybrid_clock.cc:648] HybridClock initialized: now 1782019524444834 us; error 35 us; skew 500 ppm
Jun 21 05:25:24 dist-test-slave-b357 krb5kdc[24421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019524, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:24.448220 24584 init.cc:377] Logged in from keytab as kudu/127.2.32.60@KRBTEST.COM (short username kudu)
I20260621 05:25:24.448850 24584 webserver.cc:460] Webserver started at http://127.2.32.60:36445/ using document root <none> and password file <none>
I20260621 05:25:24.449119 24584 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:24.449191 24584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:24.449342 24584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:24.450582 24584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-2/data/instance:
uuid: "88750ef45dc24bc982963d4b701434fb"
format_stamp: "Formatted at 2026-06-21 05:25:24 on dist-test-slave-b357"
server_key: "4acca34dbf670ceed5a2d69848d12fa1"
server_key_iv: "42231a294fd841a261204d15b0e3a7d2"
server_key_version: "encryptionkey@0"
I20260621 05:25:24.450999 24584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-2/wal/instance:
uuid: "88750ef45dc24bc982963d4b701434fb"
format_stamp: "Formatted at 2026-06-21 05:25:24 on dist-test-slave-b357"
server_key: "4acca34dbf670ceed5a2d69848d12fa1"
server_key_iv: "42231a294fd841a261204d15b0e3a7d2"
server_key_version: "encryptionkey@0"
I20260621 05:25:24.452623 24584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:25:24.453830 24599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:24.454149 24584 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:25:24.454350 24584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-2/wal
uuid: "88750ef45dc24bc982963d4b701434fb"
format_stamp: "Formatted at 2026-06-21 05:25:24 on dist-test-slave-b357"
server_key: "4acca34dbf670ceed5a2d69848d12fa1"
server_key_iv: "42231a294fd841a261204d15b0e3a7d2"
server_key_version: "encryptionkey@0"
I20260621 05:25:24.454452 24584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:24.466338 24577 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } attempt: 2
I20260621 05:25:24.466931 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.466771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:33391 (local address 127.0.0.1:47814)
0621 05:25:24.466843 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.466895 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260621 05:25:24.467376 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.467271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:33391 (local address 127.0.0.1:47822)
0621 05:25:24.467322 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.467348 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260621 05:25:24.467463 24577 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.60:33391: Network error: Client connection negotiation failed: client connection to 127.2.32.60:33391: connect: Connection refused (error 111)
I20260621 05:25:24.477100 24584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:24.479940 24584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:24.480119 24584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:24.484362 24584 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.60:33391
I20260621 05:25:24.484396 24651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.60:33391 every 8 connection(s)
I20260621 05:25:24.484751 24584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-2/data/info.pb
I20260621 05:25:24.485709 24652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:24.486418 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 24584
I20260621 05:25:24.486528 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782019225111293-2176-0/minicluster-data/master-2/wal/instance
I20260621 05:25:24.486701 2176 external_mini_cluster.cc:1428] Setting key 60e68967954d26c4ff88fcb262fb058b
I20260621 05:25:24.487836 24652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:25:24 dist-test-slave-b357 krb5kdc[24421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019524, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.60@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
Jun 21 05:25:24 dist-test-slave-b357 krb5kdc[24421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.60@KRBTEST.COM
I20260621 05:25:24.497588 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.488684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38172 (local address 127.2.32.62:45803)
0621 05:25:24.488740 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.488742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.488875 (+ 133us) server_negotiation.cc:407] Connection header received
0621 05:25:24.488926 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.488927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.488937 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.488994 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.489698 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.490157 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.491055 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.491292 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.491348 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.493789 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:24.493791 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:24.493791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:24.493800 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:24.495516 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.496262 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.496264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.496264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.496343 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.496680 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.496681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.496681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.496816 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:24.496916 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:24.497369 (+ 453us) server_negotiation.cc:299] Negotiation successful
0621 05:25:24.497460 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260621 05:25:24.497820 24656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.488588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:45803 (local address 127.0.0.1:38172)
0621 05:25:24.488766 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.488802 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 05:25:24.488869 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:25:24.489027 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:25:24.489029 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:25:24.489354 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:25:24.489587 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:24.489593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.490247 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.490247 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:24.490964 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:24.490966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.491393 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.491394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:24.491494 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.492146 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:25:24.492154 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:25:24.493621 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:25:24.495614 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:24.495621 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:24.495623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:24.496096 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:24.496423 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:24.496424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:24.496425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:24.496604 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:24.496953 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:25:24.496960 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:25:24.497164 (+ 204us) client_negotiation.cc:769] Sending connection context
0621 05:25:24.497304 (+ 140us) client_negotiation.cc:240] Negotiation successful
0621 05:25:24.497543 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":22,"thread_start_us":100,"threads_started":1}
I20260621 05:25:24.498167 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.487277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47834 (local address 127.2.32.60:33391)
0621 05:25:24.487463 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.487473 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.487508 (+ 35us) server_negotiation.cc:407] Connection header received
0621 05:25:24.488041 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.488049 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.488109 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.488249 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.489032 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.490212 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.490959 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.491408 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.491494 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.494410 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:24.494429 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:24.494430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:24.494437 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:24.496431 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.496872 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.496876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.496877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.496940 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.497317 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.497319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.497319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.497581 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:24.497753 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:24.497963 (+ 210us) server_negotiation.cc:299] Negotiation successful
0621 05:25:24.498060 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
I20260621 05:25:24.498999 24652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 05:25:24 dist-test-slave-b357 krb5kdc[24421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019524, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.60@KRBTEST.COM for kudu/127.2.32.61@KRBTEST.COM
Jun 21 05:25:24 dist-test-slave-b357 krb5kdc[24421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:25:24.507050 24656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.499302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:40249 (local address 127.0.0.1:59322)
0621 05:25:24.499428 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.499438 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:25:24.499484 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:25:24.499598 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:25:24.499599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:25:24.499755 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:25:24.499793 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:24.499795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.500677 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.500678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:24.501449 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:24.501451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.501949 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.501950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:24.501990 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.502469 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:25:24.502476 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:25:24.503978 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:25:24.505764 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:24.505765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:24.505766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:24.506014 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:24.506275 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:24.506276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:24.506276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:24.506345 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:24.506809 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:25:24.506810 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:25:24.506847 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 05:25:24.506917 (+ 70us) client_negotiation.cc:240] Negotiation successful
0621 05:25:24.506992 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260621 05:25:24.507251 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.499382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.2.32.61:40249)
0621 05:25:24.499441 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.499444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.499474 (+ 30us) server_negotiation.cc:407] Connection header received
0621 05:25:24.499534 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.499535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.499545 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.499595 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.499886 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.500554 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.501577 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.501872 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.501945 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.504106 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:24.504107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:24.504108 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:24.504115 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:24.505662 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.506110 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.506111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.506111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.506194 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.506475 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.506477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.506478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.506647 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:24.506804 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:24.506973 (+ 169us) server_negotiation.cc:299] Negotiation successful
0621 05:25:24.507075 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260621 05:25:24.507704 24652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:25:24.508466 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.499410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.2.32.62:45803)
0621 05:25:24.499445 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.499446 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.499453 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:25:24.499475 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.499476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.499486 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.499546 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.500089 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.500738 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.501682 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.501967 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.502028 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.504821 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:24.504823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:24.504823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:24.504829 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:24.506650 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.507314 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.507315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.507315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.507390 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.507661 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.507663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.507663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.507861 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:24.508159 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:24.508284 (+ 125us) server_negotiation.cc:299] Negotiation successful
0621 05:25:24.508383 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:25:24.511366 24502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } attempt: 2
Jun 21 05:25:24 dist-test-slave-b357 krb5kdc[24421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019524, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.60@KRBTEST.COM for kudu/127.2.32.60@KRBTEST.COM
Jun 21 05:25:24 dist-test-slave-b357 krb5kdc[24421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.61@KRBTEST.COM
I20260621 05:25:24.516374 24656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.507981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:33391 (local address 127.0.0.1:47844)
0621 05:25:24.508025 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.508035 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:25:24.508071 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:25:24.508335 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:25:24.508336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:25:24.508472 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:25:24.508523 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:24.508525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.509329 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.509330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:24.510195 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:24.510198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.510771 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.510772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:24.510852 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.511380 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:25:24.511385 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:25:24.513197 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:25:24.515235 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:24.515235 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:24.515236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:24.515455 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:24.515671 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:24.515671 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:24.515672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:24.515731 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:24.516033 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:25:24.516033 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:25:24.516070 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 05:25:24.516139 (+ 69us) client_negotiation.cc:240] Negotiation successful
0621 05:25:24.516232 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260621 05:25:24.516737 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.508053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47844 (local address 127.2.32.60:33391)
0621 05:25:24.508093 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.508095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.508103 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:25:24.508127 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.508128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.508139 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.508329 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.508622 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.509268 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.510305 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.510688 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.510756 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.513302 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:24.513303 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:24.513303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:24.513310 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:24.515082 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.515543 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.515544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.515544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.515599 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.515806 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.515807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.515807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.515946 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:24.516010 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:24.516555 (+ 545us) server_negotiation.cc:299] Negotiation successful
0621 05:25:24.516634 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Jun 21 05:25:24 dist-test-slave-b357 krb5kdc[24421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019524, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for kudu/127.2.32.60@KRBTEST.COM
I20260621 05:25:24.518443 24652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb: Bootstrap starting.
I20260621 05:25:24.519380 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.509095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59334 (local address 127.2.32.61:40249)
0621 05:25:24.509142 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.509144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.509153 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:25:24.509183 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.509184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.509192 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.509236 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.510009 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.510759 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.511752 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.512067 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.512119 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.515954 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:24.515956 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:24.515956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:24.515963 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:24.517776 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.518202 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.518203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.518203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.518262 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.518508 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.518509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.518509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.518638 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:24.518763 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:24.519007 (+ 244us) server_negotiation.cc:299] Negotiation successful
0621 05:25:24.519089 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 05:25:24.519415 24652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:24.519902 24652 log.cc:824] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:24.519997 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.511588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:33391 (local address 127.0.0.1:47846)
0621 05:25:24.511670 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.511680 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:25:24.511715 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:25:24.512453 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:25:24.512454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:25:24.512611 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:25:24.512647 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:24.512649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.514031 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.514032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:24.514797 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:24.514800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.515315 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.515316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:24.515370 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.515777 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:25:24.515782 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:25:24.517072 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:25:24.518957 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:24.518957 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:24.518958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:24.519174 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:24.519390 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:24.519391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:24.519391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:24.519444 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:24.519736 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:25:24.519736 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:25:24.519776 (+ 40us) client_negotiation.cc:769] Sending connection context
0621 05:25:24.519820 (+ 44us) client_negotiation.cc:240] Negotiation successful
0621 05:25:24.519906 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260621 05:25:24.520392 24657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.511786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47846 (local address 127.2.32.60:33391)
0621 05:25:24.512307 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.512309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.512317 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:25:24.512335 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.512336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.512355 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.512724 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.512769 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.513932 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.514892 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.515231 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.515507 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.517182 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:24.517183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:24.517183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:24.517190 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:24.518867 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.519271 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.519272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.519272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.519325 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.519511 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.519512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.519512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.519659 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:24.520159 (+ 500us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:24.520198 (+ 39us) server_negotiation.cc:299] Negotiation successful
0621 05:25:24.520271 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":95,"threads_started":1}
I20260621 05:25:24.521003 24652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb: No bootstrap required, opened a new log
I20260621 05:25:24.522326 24652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d635affcecb468fa7ada5aa32e7856c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } } peers { permanent_uuid: "a5ab9c20160842a1b50d89e2aa226773" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } } peers { permanent_uuid: "88750ef45dc24bc982963d4b701434fb" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } }
I20260621 05:25:24.522459 24652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88750ef45dc24bc982963d4b701434fb, State: Initialized, Role: FOLLOWER
I20260621 05:25:24.522521 24652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d635affcecb468fa7ada5aa32e7856c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } } peers { permanent_uuid: "a5ab9c20160842a1b50d89e2aa226773" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } } peers { permanent_uuid: "88750ef45dc24bc982963d4b701434fb" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } }
I20260621 05:25:24.522764 24659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d635affcecb468fa7ada5aa32e7856c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } } peers { permanent_uuid: "a5ab9c20160842a1b50d89e2aa226773" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } } peers { permanent_uuid: "88750ef45dc24bc982963d4b701434fb" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } } }
I20260621 05:25:24.522835 24652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:24.522835 24659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:25:24.528122 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:25:24.528915 24502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c: Bootstrap starting.
I20260621 05:25:24.529630 24502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:24.529937 24502 log.cc:824] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:24.529183 24476 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5d635affcecb468fa7ada5aa32e7856c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38178
W20260621 05:25:24.524684 24670 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260621 05:25:24.528077 24584 master_runner.cc:186] Error getting master registration for 127.2.32.62:45803: OK, instance_id { permanent_uuid: "5d635affcecb468fa7ada5aa32e7856c" instance_seqno: 1782019524162893 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260621 05:25:24.530349 24670 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:25:24.530433 24584 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 05:25:24.531097 24502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c: No bootstrap required, opened a new log
I20260621 05:25:24.532464 24502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d635affcecb468fa7ada5aa32e7856c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } } peers { permanent_uuid: "a5ab9c20160842a1b50d89e2aa226773" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } } peers { permanent_uuid: "88750ef45dc24bc982963d4b701434fb" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } }
I20260621 05:25:24.532641 24502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d635affcecb468fa7ada5aa32e7856c, State: Initialized, Role: FOLLOWER
I20260621 05:25:24.532763 24502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d635affcecb468fa7ada5aa32e7856c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } } peers { permanent_uuid: "a5ab9c20160842a1b50d89e2aa226773" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } } peers { permanent_uuid: "88750ef45dc24bc982963d4b701434fb" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } }
I20260621 05:25:24.533092 24502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:24.533052 24673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d635affcecb468fa7ada5aa32e7856c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } } peers { permanent_uuid: "a5ab9c20160842a1b50d89e2aa226773" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } } peers { permanent_uuid: "88750ef45dc24bc982963d4b701434fb" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } } }
I20260621 05:25:24.533185 24673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:25:24.533151 24476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260621 05:25:24.533274 24476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:24.535204 24476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d635affcecb468fa7ada5aa32e7856c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } } peers { permanent_uuid: "a5ab9c20160842a1b50d89e2aa226773" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } } peers { permanent_uuid: "88750ef45dc24bc982963d4b701434fb" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } }
I20260621 05:25:24.535459 24476 leader_election.cc:288] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [CANDIDATE]: Term 1 election: Requested vote from peers a5ab9c20160842a1b50d89e2aa226773 (127.2.32.61:40249), 88750ef45dc24bc982963d4b701434fb (127.2.32.60:33391)
I20260621 05:25:24.536779 24626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d635affcecb468fa7ada5aa32e7856c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "88750ef45dc24bc982963d4b701434fb"
I20260621 05:25:24.536877 24626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:24.537726 24626 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d635affcecb468fa7ada5aa32e7856c in term 1.
I20260621 05:25:24.537933 24453 leader_election.cc:302] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d635affcecb468fa7ada5aa32e7856c, 88750ef45dc24bc982963d4b701434fb; no voters:
I20260621 05:25:24.538061 24673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:25:24.538233 24673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [term 1 LEADER]: Becoming Leader. State: Replica: 5d635affcecb468fa7ada5aa32e7856c, State: Running, Role: LEADER
I20260621 05:25:24.538335 24673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d635affcecb468fa7ada5aa32e7856c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } } peers { permanent_uuid: "a5ab9c20160842a1b50d89e2aa226773" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } } peers { permanent_uuid: "88750ef45dc24bc982963d4b701434fb" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } }
I20260621 05:25:24.538754 24673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d635affcecb468fa7ada5aa32e7856c. Latest consensus state: current_term: 1 leader_uuid: "5d635affcecb468fa7ada5aa32e7856c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d635affcecb468fa7ada5aa32e7856c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } } peers { permanent_uuid: "a5ab9c20160842a1b50d89e2aa226773" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } } peers { permanent_uuid: "88750ef45dc24bc982963d4b701434fb" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } } }
I20260621 05:25:24.538832 24673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [sys.catalog]: This master's current role is: LEADER
W20260621 05:25:24.539760 24693 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:25:24.539812 24693 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:25:24.540182 24692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:25:24.540354 24692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 05:25:24.540439 24434 master_runner.cc:186] Error getting master registration for 127.2.32.61:40249: OK, instance_id { permanent_uuid: "a5ab9c20160842a1b50d89e2aa226773" instance_seqno: 1782019524319421 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 05:25:24.542662 24434 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 05:25:24.546792 24626 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb [term 1 FOLLOWER]: Refusing update from remote peer 5d635affcecb468fa7ada5aa32e7856c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:25:24.547101 24673 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [LEADER]: Connected to new peer: Peer: permanent_uuid: "88750ef45dc24bc982963d4b701434fb" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:25:24.548415 24659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d635affcecb468fa7ada5aa32e7856c. Latest consensus state: current_term: 1 leader_uuid: "5d635affcecb468fa7ada5aa32e7856c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d635affcecb468fa7ada5aa32e7856c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } } peers { permanent_uuid: "a5ab9c20160842a1b50d89e2aa226773" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } } peers { permanent_uuid: "88750ef45dc24bc982963d4b701434fb" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } } }
I20260621 05:25:24.548511 24659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:25:24.550454 24657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.536744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.2.32.60:33391)
0621 05:25:24.536782 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.536785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.540039 (+ 3254us) server_negotiation.cc:407] Connection header received
0621 05:25:24.540152 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.540153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.540165 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.540215 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.540949 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.541565 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.542599 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.542915 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.542996 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.544608 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:24.544609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:24.544609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:24.544616 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:24.547789 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.549016 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.549018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.549019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.549136 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.549471 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.549472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.549472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.549656 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:24.549809 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:24.550185 (+ 376us) server_negotiation.cc:299] Negotiation successful
0621 05:25:24.550298 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:25:24.556450 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.538380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59350 (local address 127.2.32.61:40249)
0621 05:25:24.538446 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.538448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.544912 (+ 6464us) server_negotiation.cc:407] Connection header received
0621 05:25:24.544975 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.544976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.545001 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.545060 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.546064 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.546800 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.549503 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.549858 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.549908 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.552351 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:24.552353 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:24.552353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:24.552362 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:24.554113 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.554830 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.554833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.554833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.554908 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.555148 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.555149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.555149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.555291 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:24.555407 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:24.556235 (+ 828us) server_negotiation.cc:299] Negotiation successful
0621 05:25:24.556325 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 05:25:24.558552 24577 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } attempt: 3
I20260621 05:25:24.560359 24689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5d635affcecb468fa7ada5aa32e7856c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d635affcecb468fa7ada5aa32e7856c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } } peers { permanent_uuid: "a5ab9c20160842a1b50d89e2aa226773" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } } peers { permanent_uuid: "88750ef45dc24bc982963d4b701434fb" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } } }
I20260621 05:25:24.560451 24689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:24.560493 24659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d635affcecb468fa7ada5aa32e7856c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d635affcecb468fa7ada5aa32e7856c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } } peers { permanent_uuid: "a5ab9c20160842a1b50d89e2aa226773" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } } peers { permanent_uuid: "88750ef45dc24bc982963d4b701434fb" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } } }
I20260621 05:25:24.560545 24697 mvcc.cc:205] Tried to move back new op lower bound from 7299151972540575744 to 7299151972509712384. Current Snapshot: MvccSnapshot[applied={T|T < 7299151972540575744}]
I20260621 05:25:24.560571 24659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:25:24.560720 24696 mvcc.cc:205] Tried to move back new op lower bound from 7299151972540575744 to 7299151972509712384. Current Snapshot: MvccSnapshot[applied={T|T < 7299151972540575744}]
I20260621 05:25:24.563381 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.538770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38194 (local address 127.2.32.62:45803)
0621 05:25:24.538804 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.538805 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.545728 (+ 6923us) server_negotiation.cc:407] Connection header received
0621 05:25:24.545833 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.545834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.545844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.545873 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.555265 (+ 9392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.555976 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.557317 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.557588 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.557665 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.559597 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:24.559598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:24.559599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:24.559605 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:24.562111 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.562513 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.562514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.562515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.562564 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.562760 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.562761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.562761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.562881 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:24.562962 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:24.563208 (+ 246us) server_negotiation.cc:299] Negotiation successful
0621 05:25:24.563279 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Jun 21 05:25:24 dist-test-slave-b357 krb5kdc[24421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019524, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.61@KRBTEST.COM for kudu/127.2.32.60@KRBTEST.COM
I20260621 05:25:24.567284 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.558784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:33391 (local address 127.0.0.1:47870)
0621 05:25:24.558863 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:24.558874 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 05:25:24.558911 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:25:24.559174 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:25:24.559174 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:25:24.559337 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:25:24.559380 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:24.559381 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.560511 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.560512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:24.561299 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:24.561302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.561887 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.561888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:24.561965 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.562439 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:25:24.562444 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:25:24.563978 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:25:24.566084 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:24.566085 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:24.566085 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:24.566352 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:24.566567 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:24.566568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:24.566569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:24.566633 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:24.566913 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:25:24.566914 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:25:24.566974 (+ 60us) client_negotiation.cc:769] Sending connection context
0621 05:25:24.567055 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 05:25:24.567190 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260621 05:25:24.567692 24692 catalog_manager.cc:1347] Generated new cluster ID: cccc6a74afc248b4b3805f36822f9855
I20260621 05:25:24.567747 24692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:25:24.567293 24657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:24.558980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47870 (local address 127.2.32.60:33391)
0621 05:25:24.559020 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.559022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.559029 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:25:24.559050 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.559051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.559059 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.559109 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.559464 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.560419 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.561418 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:24.561749 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:24.561813 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:24.564097 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:24.564098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:24.564098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:24.564119 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:24.565971 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.566426 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.566427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.566428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.566488 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:24.566700 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:24.566701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:24.566702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:24.566827 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:24.566906 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:24.567109 (+ 203us) server_negotiation.cc:299] Negotiation successful
0621 05:25:24.567209 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:25:24.577091 24577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773: Bootstrap starting.
I20260621 05:25:24.577906 24577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:24.578226 24577 log.cc:824] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:24.579239 24577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773: No bootstrap required, opened a new log
I20260621 05:25:24.582038 24577 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d635affcecb468fa7ada5aa32e7856c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } } peers { permanent_uuid: "a5ab9c20160842a1b50d89e2aa226773" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } } peers { permanent_uuid: "88750ef45dc24bc982963d4b701434fb" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } }
I20260621 05:25:24.582187 24577 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ab9c20160842a1b50d89e2aa226773, State: Initialized, Role: FOLLOWER
I20260621 05:25:24.582276 24577 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d635affcecb468fa7ada5aa32e7856c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } } peers { permanent_uuid: "a5ab9c20160842a1b50d89e2aa226773" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } } peers { permanent_uuid: "88750ef45dc24bc982963d4b701434fb" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } }
I20260621 05:25:24.582602 24702 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d635affcecb468fa7ada5aa32e7856c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } } peers { permanent_uuid: "a5ab9c20160842a1b50d89e2aa226773" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } } peers { permanent_uuid: "88750ef45dc24bc982963d4b701434fb" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } } }
I20260621 05:25:24.582684 24550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:24.584028 24550 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773 [term 1 FOLLOWER]: Refusing update from remote peer 5d635affcecb468fa7ada5aa32e7856c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:25:24.584352 24673 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ab9c20160842a1b50d89e2aa226773" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:25:24.582711 24551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d635affcecb468fa7ada5aa32e7856c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a5ab9c20160842a1b50d89e2aa226773"
I20260621 05:25:24.585129 24551 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d635affcecb468fa7ada5aa32e7856c in term 1.
I20260621 05:25:24.582629 24577 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:24.587569 24702 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:25:24.587821 24702 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d635affcecb468fa7ada5aa32e7856c. Latest consensus state: current_term: 1 leader_uuid: "5d635affcecb468fa7ada5aa32e7856c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d635affcecb468fa7ada5aa32e7856c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } } peers { permanent_uuid: "a5ab9c20160842a1b50d89e2aa226773" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } } peers { permanent_uuid: "88750ef45dc24bc982963d4b701434fb" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } } }
I20260621 05:25:24.587972 24702 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:25:24.588078 24702 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d635affcecb468fa7ada5aa32e7856c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d635affcecb468fa7ada5aa32e7856c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } } peers { permanent_uuid: "a5ab9c20160842a1b50d89e2aa226773" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } } peers { permanent_uuid: "88750ef45dc24bc982963d4b701434fb" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } } }
I20260621 05:25:24.588291 24702 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:25:24.592094 24704 mvcc.cc:205] Tried to move back new op lower bound from 7299151972540575744 to 7299151972509712384. Current Snapshot: MvccSnapshot[applied={T|T < 7299151972540575744}]
I20260621 05:25:24.592289 24692 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:25:24.592880 24689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5d635affcecb468fa7ada5aa32e7856c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d635affcecb468fa7ada5aa32e7856c" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 45803 } } peers { permanent_uuid: "a5ab9c20160842a1b50d89e2aa226773" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 40249 } } peers { permanent_uuid: "88750ef45dc24bc982963d4b701434fb" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 33391 } } }
I20260621 05:25:24.592968 24689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:24.592978 24692 catalog_manager.cc:1504] Loading token signing keys...
W20260621 05:25:24.593350 24717 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:25:24.593829 24717 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:25:24.596393 24692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d635affcecb468fa7ada5aa32e7856c: Generated new TSK 0
I20260621 05:25:24.596592 24692 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 05:25:24.624929 24582 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:24.622889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.2.32.61:40249)
0621 05:25:24.622934 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.622936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.623853 (+ 917us) server_negotiation.cc:407] Connection header received
0621 05:25:24.624048 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.624049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.624061 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.624093 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.624858 (+ 765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59358: BlockingRecv error: recv got EOF from 127.0.0.1:59358 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260621 05:25:24.625231 24505 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:24.623158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38200 (local address 127.2.32.62:45803)
0621 05:25:24.623197 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.623198 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.623994 (+ 796us) server_negotiation.cc:407] Connection header received
0621 05:25:24.624495 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.624496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.624506 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.624544 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.625176 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38200: BlockingRecv error: recv got EOF from 127.0.0.1:38200 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260621 05:25:24.625984 24657 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:24.623369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47884 (local address 127.2.32.60:33391)
0621 05:25:24.623407 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.623409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.625450 (+ 2041us) server_negotiation.cc:407] Connection header received
0621 05:25:24.625607 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.625608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.625618 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.625650 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.625930 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47884: BlockingRecv error: recv got EOF from 127.0.0.1:47884 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260621 05:25:24.628299 24582 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:24.627563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59366 (local address 127.2.32.61:40249)
0621 05:25:24.627594 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.627596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.627818 (+ 222us) server_negotiation.cc:407] Connection header received
0621 05:25:24.627917 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.627917 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.627932 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.627959 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.628256 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59366: BlockingRecv error: recv got EOF from 127.0.0.1:59366 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260621 05:25:24.631171 24582 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:24.630458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59370 (local address 127.2.32.61:40249)
0621 05:25:24.630487 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.630489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.630692 (+ 203us) server_negotiation.cc:407] Connection header received
0621 05:25:24.630780 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.630781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.630789 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.630816 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.631099 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59370: BlockingRecv error: recv got EOF from 127.0.0.1:59370 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260621 05:25:24.634986 24582 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:24.634283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59374 (local address 127.2.32.61:40249)
0621 05:25:24.634313 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.634314 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.634494 (+ 180us) server_negotiation.cc:407] Connection header received
0621 05:25:24.634640 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.634641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.634650 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.634677 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.634948 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59374: BlockingRecv error: recv got EOF from 127.0.0.1:59374 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260621 05:25:24.640763 24582 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:24.640083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59384 (local address 127.2.32.61:40249)
0621 05:25:24.640114 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.640116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.640299 (+ 183us) server_negotiation.cc:407] Connection header received
0621 05:25:24.640388 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.640388 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.640397 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.640424 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.640722 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59384: BlockingRecv error: recv got EOF from 127.0.0.1:59384 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260621 05:25:24.650535 24582 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:24.649829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.2.32.61:40249)
0621 05:25:24.649861 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.649863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.650053 (+ 190us) server_negotiation.cc:407] Connection header received
0621 05:25:24.650142 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.650143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.650152 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.650179 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.650470 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59394: BlockingRecv error: recv got EOF from 127.0.0.1:59394 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260621 05:25:24.668732 24582 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:24.667887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59402 (local address 127.2.32.61:40249)
0621 05:25:24.667929 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.667931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.668176 (+ 245us) server_negotiation.cc:407] Connection header received
0621 05:25:24.668276 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.668277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.668289 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.668318 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.668674 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59402: BlockingRecv error: recv got EOF from 127.0.0.1:59402 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260621 05:25:24.703096 24582 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:24.702155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59406 (local address 127.2.32.61:40249)
0621 05:25:24.702204 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.702206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.702463 (+ 257us) server_negotiation.cc:407] Connection header received
0621 05:25:24.702575 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.702576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.702589 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.702620 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.703028 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59406: BlockingRecv error: recv got EOF from 127.0.0.1:59406 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260621 05:25:24.770200 24582 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:24.768926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.2.32.61:40249)
0621 05:25:24.769039 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.769042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.769097 (+ 55us) server_negotiation.cc:407] Connection header received
0621 05:25:24.769271 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.769273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.769296 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.769368 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.770121 (+ 753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59414: BlockingRecv error: recv got EOF from 127.0.0.1:59414 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260621 05:25:24.900274 24582 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:24.899332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.2.32.61:40249)
0621 05:25:24.899491 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:24.899497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:24.899515 (+ 18us) server_negotiation.cc:407] Connection header received
0621 05:25:24.899589 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:24.899590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:24.899605 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:24.899703 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:24.900197 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59418: BlockingRecv error: recv got EOF from 127.0.0.1:59418 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260621 05:25:25.158577 24582 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:25.157600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.2.32.61:40249)
0621 05:25:25.157687 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:25.157689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:25.157700 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:25:25.157779 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:25.157780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:25.157791 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:25.157868 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:25.158493 (+ 625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59432: BlockingRecv error: recv got EOF from 127.0.0.1:59432 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 05:25:25.530975 24670 catalog_manager.cc:1259] Loaded cluster ID: cccc6a74afc248b4b3805f36822f9855
I20260621 05:25:25.531106 24670 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb: loading cluster ID for follower catalog manager: success
I20260621 05:25:25.532073 24670 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb: acquiring CA information for follower catalog manager: success
I20260621 05:25:25.532436 24670 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 88750ef45dc24bc982963d4b701434fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:25:25.594295 24717 catalog_manager.cc:1259] Loaded cluster ID: cccc6a74afc248b4b3805f36822f9855
I20260621 05:25:25.594383 24717 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773: loading cluster ID for follower catalog manager: success
I20260621 05:25:25.595311 24717 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773: acquiring CA information for follower catalog manager: success
I20260621 05:25:25.595638 24717 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a5ab9c20160842a1b50d89e2aa226773: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:25:25.674862 24785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:25.672036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.2.32.61:40249)
0621 05:25:25.672203 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:25.672206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:25.672301 (+ 95us) server_negotiation.cc:407] Connection header received
0621 05:25:25.672409 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:25.672410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:25.672477 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:25.672578 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:25:25.672733 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:25.673257 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:25.674025 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:25.674230 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:25.674305 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:25.674493 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:25:25.674602 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:25:25.674694 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:25.674734 (+ 40us) server_negotiation.cc:299] Negotiation successful
0621 05:25:25.674754 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":73,"threads_started":1}
I20260621 05:25:25.679385 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:25.676852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47888 (local address 127.2.32.60:33391)
0621 05:25:25.676947 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:25.676949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:25.676957 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:25:25.677009 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:25.677010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:25.677041 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:25.677115 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:25:25.677286 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:25.677753 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:25.678484 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:25.678685 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:25.678751 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:25.678887 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:25:25.678974 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:25:25.679030 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:25.679109 (+ 79us) server_negotiation.cc:299] Negotiation successful
0621 05:25:25.679265 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":48,"threads_started":1}
I20260621 05:25:25.683887 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:25.681349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47898 (local address 127.2.32.60:33391)
0621 05:25:25.681386 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:25.681387 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:25.681395 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:25:25.681416 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:25.681417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:25.681426 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:25.681470 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:25:25.681630 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:25.682337 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:25.683226 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:25.683429 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:25.683500 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:25.683677 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:25:25.683705 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:25:25.683751 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:25.683802 (+ 51us) server_negotiation.cc:299] Negotiation successful
0621 05:25:25.683819 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260621 05:25:25.684537 24785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:25.681171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.2.32.61:40249)
0621 05:25:25.681212 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:25.681214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:25.681529 (+ 315us) server_negotiation.cc:407] Connection header received
0621 05:25:25.681585 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:25.681586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:25.681599 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:25.681678 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:25:25.681837 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:25.682522 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:25.683682 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:25.684044 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:25.684104 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:25.684292 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:25:25.684321 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:25:25.684377 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:25.684449 (+ 72us) server_negotiation.cc:299] Negotiation successful
0621 05:25:25.684475 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260621 05:25:25.684582 24799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:25.681255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38204 (local address 127.2.32.62:45803)
0621 05:25:25.681403 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:25.681405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:25.681416 (+ 11us) server_negotiation.cc:407] Connection header received
0621 05:25:25.681501 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:25.681502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:25.681551 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:25.681627 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:25:25.681776 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:25.682323 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:25.683614 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:25.683907 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:25.683970 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:25.684195 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:25:25.684315 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:25:25.684357 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:25.684484 (+ 127us) server_negotiation.cc:299] Negotiation successful
0621 05:25:25.684505 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":58,"threads_started":1}
I20260621 05:25:25.687877 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 24434
I20260621 05:25:25.694729 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 24509
I20260621 05:25:25.700886 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 24584
2026-06-21T05:25:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2857 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 05:25:25 dist-test-slave-b357 krb5kdc[24802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 05:25:25 dist-test-slave-b357 krb5kdc[24802](info): set up 2 sockets
Jun 21 05:25:25 dist-test-slave-b357 krb5kdc[24802](info): commencing operation
krb5kdc: starting...
W20260621 05:25:26.764019 2176 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/test-user.keytab.
Jun 21 05:25:26 dist-test-slave-b357 krb5kdc[24802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T05:25:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T05:25:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.32.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
Entry for principal kudu/127.2.32.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab.
I20260621 05:25:26.890771 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:38777
--webserver_interface=127.2.32.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:38777,127.2.32.61:42369,127.2.32.60:34995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 05:25:26.973881 24815 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 05:25:26.974107 24815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:26.974123 24815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:26.974143 24815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:26.975284 24815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:26.975325 24815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:26.975339 24815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:26.975353 24815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:26.975365 24815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:26.978034 24815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:38777,127.2.32.61:42369,127.2.32.60:34995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.62.keytab
--principal=kudu/127.2.32.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.62:38777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.32.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:26.978403 24815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:26.978729 24815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:26.984650 24821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:26.984714 24823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:26.984714 24820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:26.985095 24815 server_base.cc:1034] running on GCE node
I20260621 05:25:26.985473 24815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:26.985822 24815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:26.987030 24815 hybrid_clock.cc:648] HybridClock initialized: now 1782019526986992 us; error 58 us; skew 500 ppm
Jun 21 05:25:26 dist-test-slave-b357 krb5kdc[24802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019526, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:26.990027 24815 init.cc:377] Logged in from keytab as kudu/127.2.32.62@KRBTEST.COM (short username kudu)
I20260621 05:25:26.990520 24815 webserver.cc:460] Webserver started at http://127.2.32.62:45177/ using document root <none> and password file <none>
I20260621 05:25:26.990782 24815 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:26.990833 24815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:26.990959 24815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:26.992065 24815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-0/data/instance:
uuid: "8f7054764766468c8fc78671b52824a0"
format_stamp: "Formatted at 2026-06-21 05:25:26 on dist-test-slave-b357"
server_key: "7e6fb76b2b0da97cec0caf311c5a68c7"
server_key_iv: "e27b43905aa290601998b9da03813c6a"
server_key_version: "encryptionkey@0"
I20260621 05:25:26.992424 24815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-0/wal/instance:
uuid: "8f7054764766468c8fc78671b52824a0"
format_stamp: "Formatted at 2026-06-21 05:25:26 on dist-test-slave-b357"
server_key: "7e6fb76b2b0da97cec0caf311c5a68c7"
server_key_iv: "e27b43905aa290601998b9da03813c6a"
server_key_version: "encryptionkey@0"
I20260621 05:25:26.993886 24815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:25:26.995056 24830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:26.995368 24815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:25:26.995501 24815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-0/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-0/wal
uuid: "8f7054764766468c8fc78671b52824a0"
format_stamp: "Formatted at 2026-06-21 05:25:26 on dist-test-slave-b357"
server_key: "7e6fb76b2b0da97cec0caf311c5a68c7"
server_key_iv: "e27b43905aa290601998b9da03813c6a"
server_key_version: "encryptionkey@0"
I20260621 05:25:26.995581 24815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:27.017897 24815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:27.020332 24815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:27.020510 24815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:27.025187 24815 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.62:38777
I20260621 05:25:27.025234 24882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.62:38777 every 8 connection(s)
I20260621 05:25:27.025589 24815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-0/data/info.pb
I20260621 05:25:27.026648 24883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:27.027144 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 24815
I20260621 05:25:27.027266 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-0/wal/instance
I20260621 05:25:27.027448 2176 external_mini_cluster.cc:1428] Setting key 54459d4101278356c626851b367042ed
I20260621 05:25:27.028555 24883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.32.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.61@KRBTEST.COM" created.
Jun 21 05:25:27 dist-test-slave-b357 krb5kdc[24802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019526, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:25:27.046016 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.029311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:38777 (local address 127.0.0.1:37016)
0621 05:25:27.029595 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.029670 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0621 05:25:27.030812 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:25:27.031297 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:25:27.031301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:25:27.031803 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:25:27.032098 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:27.032105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.033236 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.033237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:27.034738 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:27.034743 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.035433 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.035435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:27.035531 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.036179 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:25:27.036190 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:25:27.041045 (+ 4855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:25:27.044166 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:27.044172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:27.044174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:27.044394 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:27.044808 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:27.044810 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:27.044811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:27.044969 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:27.045534 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:25:27.045541 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:25:27.045605 (+ 64us) client_negotiation.cc:769] Sending connection context
0621 05:25:27.045689 (+ 84us) client_negotiation.cc:240] Negotiation successful
0621 05:25:27.045836 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":128,"threads_started":1}
I20260621 05:25:27.046105 24887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.029508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37016 (local address 127.2.32.62:38777)
0621 05:25:27.029739 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.029753 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.029792 (+ 39us) server_negotiation.cc:407] Connection header received
0621 05:25:27.031041 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.031044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.031064 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.031260 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.032243 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.033106 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.034893 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.035312 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.035401 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.041153 (+ 5752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:27.041159 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:27.041160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:27.041166 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:27.044062 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.044492 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.044495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.044496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.044625 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.045067 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.045069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.045069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.045389 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:27.045492 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:27.045749 (+ 257us) server_negotiation.cc:299] Negotiation successful
0621 05:25:27.045926 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":173,"thread_start_us":163,"threads_started":1}
I20260621 05:25:27.047118 24883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:25:27.047724 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.047558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:42369 (local address 127.0.0.1:38322)
0621 05:25:27.047636 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.047696 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.61:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260621 05:25:27.047930 24831 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.32.61:42369: connect: Connection refused (error 111)
I20260621 05:25:27.048489 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.048373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:42369 (local address 127.0.0.1:38324)
0621 05:25:27.048428 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.048459 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.61:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260621 05:25:27.048657 24883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.61:42369: Network error: Client connection negotiation failed: client connection to 127.2.32.61:42369: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.61.keytab.
Entry for principal kudu/127.2.32.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.61.keytab.
I20260621 05:25:27.058835 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.61:42369
--webserver_interface=127.2.32.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.61.keytab
--principal=kudu/127.2.32.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:38777,127.2.32.61:42369,127.2.32.60:34995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:25:27.097887 24883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } attempt: 1
I20260621 05:25:27.098694 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.098471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:42369 (local address 127.0.0.1:38334)
0621 05:25:27.098589 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.098648 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.61:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260621 05:25:27.099364 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.099241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:42369 (local address 127.0.0.1:38338)
0621 05:25:27.099295 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.099327 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.61:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260621 05:25:27.099473 24883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.61:42369: Network error: Client connection negotiation failed: client connection to 127.2.32.61:42369: connect: Connection refused (error 111)
W20260621 05:25:27.141646 24890 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 05:25:27.141913 24890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:27.141945 24890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:27.141993 24890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:27.143261 24890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:27.143322 24890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:27.143349 24890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:27.143381 24890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:27.143412 24890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:27.144970 24890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:38777,127.2.32.61:42369,127.2.32.60:34995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.61.keytab
--principal=kudu/127.2.32.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.61:42369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.32.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:27.145255 24890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:27.145499 24890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:27.150638 24896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:27.150728 24895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:27.150744 24898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:27.150789 24890 server_base.cc:1034] running on GCE node
I20260621 05:25:27.151283 24890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:27.151616 24890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:27.152771 24890 hybrid_clock.cc:648] HybridClock initialized: now 1782019527152738 us; error 55 us; skew 500 ppm
Jun 21 05:25:27 dist-test-slave-b357 krb5kdc[24802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019527, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:27.155464 24890 init.cc:377] Logged in from keytab as kudu/127.2.32.61@KRBTEST.COM (short username kudu)
I20260621 05:25:27.155902 24890 webserver.cc:460] Webserver started at http://127.2.32.61:41065/ using document root <none> and password file <none>
I20260621 05:25:27.156138 24890 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:27.156183 24890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:27.156351 24890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:27.157752 24890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-1/data/instance:
uuid: "a1fd9327993f438d841f13ab42b77bab"
format_stamp: "Formatted at 2026-06-21 05:25:27 on dist-test-slave-b357"
server_key: "83c229417cf5b641939f4a22af7533bd"
server_key_iv: "19e26e123b218a593c728a9f244bcf84"
server_key_version: "encryptionkey@0"
I20260621 05:25:27.158242 24890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-1/wal/instance:
uuid: "a1fd9327993f438d841f13ab42b77bab"
format_stamp: "Formatted at 2026-06-21 05:25:27 on dist-test-slave-b357"
server_key: "83c229417cf5b641939f4a22af7533bd"
server_key_iv: "19e26e123b218a593c728a9f244bcf84"
server_key_version: "encryptionkey@0"
I20260621 05:25:27.160274 24890 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260621 05:25:27.161327 24905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:27.161623 24890 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:25:27.161805 24890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-1/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-1/wal
uuid: "a1fd9327993f438d841f13ab42b77bab"
format_stamp: "Formatted at 2026-06-21 05:25:27 on dist-test-slave-b357"
server_key: "83c229417cf5b641939f4a22af7533bd"
server_key_iv: "19e26e123b218a593c728a9f244bcf84"
server_key_version: "encryptionkey@0"
I20260621 05:25:27.161922 24890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:27.167631 24883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } attempt: 2
I20260621 05:25:27.168085 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.167956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:42369 (local address 127.0.0.1:38354)
0621 05:25:27.168014 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.168054 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.61:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260621 05:25:27.168417 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.168340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:42369 (local address 127.0.0.1:38362)
0621 05:25:27.168380 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.168402 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.61:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260621 05:25:27.168540 24883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.61:42369: Network error: Client connection negotiation failed: client connection to 127.2.32.61:42369: connect: Connection refused (error 111)
I20260621 05:25:27.179422 24890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:27.181922 24890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:27.182078 24890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:27.187547 24890 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.61:42369
I20260621 05:25:27.187589 24957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.61:42369 every 8 connection(s)
I20260621 05:25:27.188009 24890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-1/data/info.pb
I20260621 05:25:27.189066 24958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:27.190647 24958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:25:27.194512 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 24890
I20260621 05:25:27.194600 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-1/wal/instance
I20260621 05:25:27.194814 2176 external_mini_cluster.cc:1428] Setting key a9e8036b56df9c6bb9b56008855f1997
Jun 21 05:25:27 dist-test-slave-b357 krb5kdc[24802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019527, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.61@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:25:27.201526 24887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.191496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37028 (local address 127.2.32.62:38777)
0621 05:25:27.191590 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.191593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.191768 (+ 175us) server_negotiation.cc:407] Connection header received
0621 05:25:27.192226 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.192228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.192243 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.192328 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.192943 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.193491 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.194963 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.195294 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.195391 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.198475 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:27.198477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:27.198477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:27.198486 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:27.200119 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.200562 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.200563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.200564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.200630 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.200874 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.200875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.200875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.200996 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:27.201072 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:27.201349 (+ 277us) server_negotiation.cc:299] Negotiation successful
0621 05:25:27.201435 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 05:25:27.201733 24960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.191428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:38777 (local address 127.0.0.1:37028)
0621 05:25:27.191597 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.191646 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 05:25:27.192092 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:25:27.192398 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:25:27.192401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:25:27.192646 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:25:27.192831 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:27.192835 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.193638 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.193640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:27.194810 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:27.194814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.195415 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.195416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:27.195585 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.196242 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:25:27.196256 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:25:27.198350 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:25:27.200229 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:27.200235 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:27.200238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:27.200471 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:27.200700 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:27.200701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:27.200702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:27.200806 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:27.201064 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:25:27.201070 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:25:27.201217 (+ 147us) client_negotiation.cc:769] Sending connection context
0621 05:25:27.201319 (+ 102us) client_negotiation.cc:240] Negotiation successful
0621 05:25:27.201492 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260621 05:25:27.202325 24958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.32.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.32.60@KRBTEST.COM" created.
Jun 21 05:25:27 dist-test-slave-b357 krb5kdc[24802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019527, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.61@KRBTEST.COM for kudu/127.2.32.61@KRBTEST.COM
I20260621 05:25:27.210594 24960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.202572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:42369 (local address 127.0.0.1:38366)
0621 05:25:27.202625 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.202635 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:25:27.202675 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:25:27.202948 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:25:27.202948 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:25:27.203087 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:25:27.203183 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:27.203185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.204062 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.204063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:27.204556 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:27.204558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.205063 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.205064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:27.205128 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.205631 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:25:27.205635 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:25:27.206808 (+ 1173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:25:27.208795 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:27.208798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:27.208799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:27.209127 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:27.209553 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:27.209554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:27.209555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:27.209656 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:27.210229 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:25:27.210230 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:25:27.210273 (+ 43us) client_negotiation.cc:769] Sending connection context
0621 05:25:27.210347 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 05:25:27.210464 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260621 05:25:27.210985 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.202657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38366 (local address 127.2.32.61:42369)
0621 05:25:27.202816 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.202820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.202835 (+ 15us) server_negotiation.cc:407] Connection header received
0621 05:25:27.202851 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.202852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.202862 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.202935 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.203251 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.203969 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.204658 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.204977 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.205065 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.206887 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:27.206894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:27.206895 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:27.206902 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:27.208673 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.209270 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.209274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.209274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.209444 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.209782 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.209784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.209784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.210092 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:27.210200 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:27.210782 (+ 582us) server_negotiation.cc:299] Negotiation successful
0621 05:25:27.210872 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260621 05:25:27.211659 24958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:25:27.212059 24960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.211931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:34995 (local address 127.0.0.1:55278)
0621 05:25:27.211985 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.212034 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:34995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260621 05:25:27.212189 24906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.32.60:34995: connect: Connection refused (error 111)
I20260621 05:25:27.212728 24960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.212632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:34995 (local address 127.0.0.1:55284)
0621 05:25:27.212684 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.212707 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:34995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260621 05:25:27.212850 24958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.60:34995: Network error: Client connection negotiation failed: client connection to 127.2.32.60:34995: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.32.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.60.keytab.
Entry for principal kudu/127.2.32.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.60.keytab.
I20260621 05:25:27.226323 2176 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJjmGzv/build/release/bin/kudu
/tmp/dist-test-taskJjmGzv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.60:34995
--webserver_interface=127.2.32.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.60.keytab
--principal=kudu/127.2.32.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.32.20:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.32.62:38777,127.2.32.61:42369,127.2.32.60:34995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 05:25:27.259747 24883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } attempt: 3
I20260621 05:25:27.262049 24958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } attempt: 1
I20260621 05:25:27.262446 24960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.262290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:34995 (local address 127.0.0.1:55300)
0621 05:25:27.262350 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.262397 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:34995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260621 05:25:27.262985 24960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.262878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:34995 (local address 127.0.0.1:55308)
0621 05:25:27.262930 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.262955 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:34995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260621 05:25:27.263141 24958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.60:34995: Network error: Client connection negotiation failed: client connection to 127.2.32.60:34995: connect: Connection refused (error 111)
Jun 21 05:25:27 dist-test-slave-b357 krb5kdc[24802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019526, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for kudu/127.2.32.61@KRBTEST.COM
I20260621 05:25:27.267810 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.260231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38382 (local address 127.2.32.61:42369)
0621 05:25:27.260305 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.260307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.260319 (+ 12us) server_negotiation.cc:407] Connection header received
0621 05:25:27.260372 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.260374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.260385 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.260451 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.260783 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.261282 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.261988 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.262318 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.262387 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.264973 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:27.264975 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:27.264975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:27.264985 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:27.266595 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.266991 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.266993 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.266993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.267053 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.267285 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.267285 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.267286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.267403 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:27.267488 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:27.267646 (+ 158us) server_negotiation.cc:299] Negotiation successful
0621 05:25:27.267724 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 05:25:27.267817 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.260141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:42369 (local address 127.0.0.1:38382)
0621 05:25:27.260213 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.260226 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 05:25:27.260307 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:25:27.260472 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:25:27.260472 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:25:27.260636 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:25:27.260701 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:27.260703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.261373 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.261375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:27.261904 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:27.261905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.262400 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.262401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:27.262481 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.263194 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:25:27.263200 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:25:27.264819 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:25:27.266698 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:27.266699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:27.266699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:27.266887 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:27.267156 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:27.267157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:27.267157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:27.267216 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:27.267478 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:25:27.267479 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:25:27.267525 (+ 46us) client_negotiation.cc:769] Sending connection context
0621 05:25:27.267620 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 05:25:27.267722 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260621 05:25:27.268174 24883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:25:27.268512 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.268405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:34995 (local address 127.0.0.1:55320)
0621 05:25:27.268459 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.268485 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:34995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260621 05:25:27.268815 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.268729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:34995 (local address 127.0.0.1:55324)
0621 05:25:27.268774 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.268796 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:34995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260621 05:25:27.268931 24883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.60:34995: Network error: Client connection negotiation failed: client connection to 127.2.32.60:34995: connect: Connection refused (error 111)
I20260621 05:25:27.300092 24883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } attempt: 1
I20260621 05:25:27.300841 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.300516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:34995 (local address 127.0.0.1:55340)
0621 05:25:27.300714 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.300792 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:34995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260621 05:25:27.301522 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.301190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:34995 (local address 127.0.0.1:55346)
0621 05:25:27.301449 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.301484 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:34995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":204,"threads_started":1}
W20260621 05:25:27.301642 24883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.60:34995: Network error: Client connection negotiation failed: client connection to 127.2.32.60:34995: connect: Connection refused (error 111)
I20260621 05:25:27.331465 24958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } attempt: 2
I20260621 05:25:27.332026 24960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.331876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:34995 (local address 127.0.0.1:55352)
0621 05:25:27.331947 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.331994 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:34995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 05:25:27.332520 24960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.332408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:34995 (local address 127.0.0.1:55366)
0621 05:25:27.332460 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.332500 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.32.60:34995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260621 05:25:27.332325 24965 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 05:25:27.332613 24965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 05:25:27.332630 24958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.32.60:34995: Network error: Client connection negotiation failed: client connection to 127.2.32.60:34995: connect: Connection refused (error 111)
W20260621 05:25:27.332666 24965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 05:25:27.332706 24965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 05:25:27.334978 24965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 05:25:27.335062 24965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 05:25:27.335093 24965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 05:25:27.335119 24965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 05:25:27.335160 24965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 05:25:27.338243 24965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.32.20:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.32.62:38777,127.2.32.61:42369,127.2.32.60:34995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/krb5kdc/kudu_127.2.32.60.keytab
--principal=kudu/127.2.32.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.32.60:34995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.32.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 21 Jun 2026 04:16:16 UTC on ccycloud.quasar-xycaha.root.comops.site
build id 4457881
I20260621 05:25:27.338539 24965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 05:25:27.338810 24965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 05:25:27.344038 24974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:27.344038 24971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 05:25:27.344054 24972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 05:25:27.344349 24965 server_base.cc:1034] running on GCE node
I20260621 05:25:27.344632 24965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 05:25:27.344985 24965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 05:25:27.346180 24965 hybrid_clock.cc:648] HybridClock initialized: now 1782019527346155 us; error 65 us; skew 500 ppm
Jun 21 05:25:27 dist-test-slave-b357 krb5kdc[24802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019527, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 05:25:27.349493 24965 init.cc:377] Logged in from keytab as kudu/127.2.32.60@KRBTEST.COM (short username kudu)
I20260621 05:25:27.349901 24965 webserver.cc:460] Webserver started at http://127.2.32.60:36593/ using document root <none> and password file <none>
I20260621 05:25:27.350142 24965 fs_manager.cc:359] Metadata directory not provided
I20260621 05:25:27.350219 24965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 05:25:27.350371 24965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 05:25:27.351476 24965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-2/data/instance:
uuid: "951c5567bfc444d3961766f841f92b92"
format_stamp: "Formatted at 2026-06-21 05:25:27 on dist-test-slave-b357"
server_key: "43f3498e7dbca1978f74c91e2023bbde"
server_key_iv: "53a8d44054d95daa24079263fe5405b3"
server_key_version: "encryptionkey@0"
I20260621 05:25:27.351819 24965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-2/wal/instance:
uuid: "951c5567bfc444d3961766f841f92b92"
format_stamp: "Formatted at 2026-06-21 05:25:27 on dist-test-slave-b357"
server_key: "43f3498e7dbca1978f74c91e2023bbde"
server_key_iv: "53a8d44054d95daa24079263fe5405b3"
server_key_version: "encryptionkey@0"
I20260621 05:25:27.353279 24965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 05:25:27.354413 24981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 05:25:27.354650 24965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 05:25:27.354772 24965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-2/data,/tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-2/wal
uuid: "951c5567bfc444d3961766f841f92b92"
format_stamp: "Formatted at 2026-06-21 05:25:27 on dist-test-slave-b357"
server_key: "43f3498e7dbca1978f74c91e2023bbde"
server_key_iv: "53a8d44054d95daa24079263fe5405b3"
server_key_version: "encryptionkey@0"
I20260621 05:25:27.354866 24965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 05:25:27.368237 24965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 05:25:27.370476 24965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 05:25:27.370641 24965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 05:25:27.376061 24965 rpc_server.cc:307] RPC server started. Bound to: 127.2.32.60:34995
I20260621 05:25:27.376122 25033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.32.60:34995 every 8 connection(s)
I20260621 05:25:27.376546 24965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-2/data/info.pb
I20260621 05:25:27.376868 24883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } attempt: 2
I20260621 05:25:27.378329 25034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 05:25:27.381513 25034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:25:27.382176 2176 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJjmGzv/build/release/bin/kudu as pid 24965
I20260621 05:25:27.382285 2176 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJjmGzv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782019225111293-2176-0/minicluster-data/master-2/wal/instance
I20260621 05:25:27.382514 2176 external_mini_cluster.cc:1428] Setting key 69d963a457968bbda55ee3340a0991f4
Jun 21 05:25:27 dist-test-slave-b357 krb5kdc[24802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019526, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.62@KRBTEST.COM for kudu/127.2.32.60@KRBTEST.COM
I20260621 05:25:27.387313 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.377202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:34995 (local address 127.0.0.1:55374)
0621 05:25:27.377258 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.377270 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 05:25:27.377312 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:25:27.378323 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:25:27.378324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:25:27.378514 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:25:27.378554 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:27.378555 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.379764 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.379765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:27.380572 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:27.380578 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.381409 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.381410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:27.381474 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.381900 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:25:27.381905 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:25:27.383680 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:25:27.385953 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:27.385954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:27.385955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:27.386182 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:27.386456 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:27.386457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:27.386457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:27.386512 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:27.386958 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:25:27.386959 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:25:27.387005 (+ 46us) client_negotiation.cc:769] Sending connection context
0621 05:25:27.387053 (+ 48us) client_negotiation.cc:240] Negotiation successful
0621 05:25:27.387187 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260621 05:25:27.387941 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.377444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55374 (local address 127.2.32.60:34995)
0621 05:25:27.377651 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.377662 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.377698 (+ 36us) server_negotiation.cc:407] Connection header received
0621 05:25:27.378137 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.378143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.378188 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.378345 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.378660 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.379674 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.380909 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.381318 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.381451 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.383804 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:27.383808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:27.383809 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:27.383819 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:27.385863 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.386270 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.386273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.386274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.386388 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.386576 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.386577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.386577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.386871 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:27.387513 (+ 642us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:27.387588 (+ 75us) server_negotiation.cc:299] Negotiation successful
0621 05:25:27.387737 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":123,"threads_started":1}
Jun 21 05:25:27 dist-test-slave-b357 krb5kdc[24802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019527, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.60@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
Jun 21 05:25:27 dist-test-slave-b357 krb5kdc[24802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.60@KRBTEST.COM
I20260621 05:25:27.389832 24883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0: Bootstrap starting.
I20260621 05:25:27.390882 24883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:27.391259 24883 log.cc:824] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:27.392231 24883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0: No bootstrap required, opened a new log
I20260621 05:25:27.392283 25037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.382348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.62:38777 (local address 127.0.0.1:37040)
0621 05:25:27.382785 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.382828 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 05:25:27.382892 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:25:27.383360 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:25:27.383363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:25:27.383725 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:25:27.383819 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:27.383821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.385314 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.385315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:27.386226 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:27.386228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.386726 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.386727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:27.386811 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.387504 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:25:27.387509 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:25:27.389086 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:25:27.390946 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:27.390960 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:27.390963 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:27.391286 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:27.391526 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:27.391527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:27.391527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:27.391592 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:27.391892 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:25:27.391898 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:25:27.391964 (+ 66us) client_negotiation.cc:769] Sending connection context
0621 05:25:27.392065 (+ 101us) client_negotiation.cc:240] Negotiation successful
0621 05:25:27.392169 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":111,"threads_started":1}
I20260621 05:25:27.392306 24887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.383160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37040 (local address 127.2.32.62:38777)
0621 05:25:27.383203 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.383205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.383212 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:25:27.383231 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.383231 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.383241 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.383334 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.383905 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.385213 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.386313 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.386618 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.386656 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.389182 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:27.389183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:27.389183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:27.389189 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:27.390856 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.391384 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.391385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.391386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.391447 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.391675 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.391676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.391677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.391812 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:27.391899 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:27.392092 (+ 193us) server_negotiation.cc:299] Negotiation successful
0621 05:25:27.392166 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:25:27.392701 25034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:25:27.394305 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.383009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55384 (local address 127.2.32.60:34995)
0621 05:25:27.383184 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.383186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.383382 (+ 196us) server_negotiation.cc:407] Connection header received
0621 05:25:27.383465 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.383466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.383478 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.383541 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.384410 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.385238 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.386483 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.386896 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.386973 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.390117 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:27.390119 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:27.390119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:27.390126 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:27.393046 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.393427 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.393428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.393428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.393480 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.393681 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.393682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.393682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.393798 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:27.393914 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:27.394116 (+ 202us) server_negotiation.cc:299] Negotiation successful
0621 05:25:27.394193 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":153,"thread_start_us":84,"threads_started":1}
I20260621 05:25:27.394610 24883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7054764766468c8fc78671b52824a0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } } peers { permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } } peers { permanent_uuid: "951c5567bfc444d3961766f841f92b92" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } }
I20260621 05:25:27.394754 24883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f7054764766468c8fc78671b52824a0, State: Initialized, Role: FOLLOWER
I20260621 05:25:27.394842 24883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7054764766468c8fc78671b52824a0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } } peers { permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } } peers { permanent_uuid: "951c5567bfc444d3961766f841f92b92" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } }
I20260621 05:25:27.395105 25041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7054764766468c8fc78671b52824a0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } } peers { permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } } peers { permanent_uuid: "951c5567bfc444d3961766f841f92b92" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } } }
I20260621 05:25:27.395205 25041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:25:27.395218 24883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [sys.catalog]: configured and running, proceeding with master startup.
Jun 21 05:25:27 dist-test-slave-b357 krb5kdc[24802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019527, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.60@KRBTEST.COM for kudu/127.2.32.61@KRBTEST.COM
W20260621 05:25:27.396999 25052 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:25:27.397042 25052 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 21 05:25:27 dist-test-slave-b357 krb5kdc[24802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.62@KRBTEST.COM
I20260621 05:25:27.400367 25037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.392971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.61:42369 (local address 127.0.0.1:38394)
0621 05:25:27.393012 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.393022 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:25:27.393057 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:25:27.393177 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:25:27.393177 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:25:27.393310 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:25:27.393337 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:27.393338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.393941 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.393942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:27.394735 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:27.394740 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.395337 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.395338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:27.395404 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.395845 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:25:27.395849 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:25:27.397142 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:25:27.399284 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:27.399285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:27.399285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:27.399515 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:27.399748 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:27.399749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:27.399749 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:27.399810 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:27.400077 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:25:27.400078 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:25:27.400119 (+ 41us) client_negotiation.cc:769] Sending connection context
0621 05:25:27.400179 (+ 60us) client_negotiation.cc:240] Negotiation successful
0621 05:25:27.400272 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260621 05:25:27.397869 24815 master_runner.cc:186] Error getting master registration for 127.2.32.61:42369: OK, instance_id { permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" instance_seqno: 1782019527186096 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 05:25:27.400807 24815 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 05:25:27.401011 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.393028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38394 (local address 127.2.32.61:42369)
0621 05:25:27.393071 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.393073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.393081 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:25:27.393106 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.393107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.393116 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.393172 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.393418 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.393861 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.394850 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.395218 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.395260 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.397229 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:27.397230 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:27.397231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:27.397237 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:27.399196 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.399621 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.399622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.399623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.399674 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.399875 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.399876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.399876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.399993 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:27.400051 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:27.400552 (+ 501us) server_negotiation.cc:299] Negotiation successful
0621 05:25:27.400639 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260621 05:25:27.401602 25034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } has no permanent_uuid. Determining permanent_uuid...
I20260621 05:25:27.403301 24887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.394844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37044 (local address 127.2.32.62:38777)
0621 05:25:27.394874 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.394875 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.395025 (+ 150us) server_negotiation.cc:407] Connection header received
0621 05:25:27.395048 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.395049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.395058 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.395783 (+ 725us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.395813 (+ 30us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.396511 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.397453 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.397766 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.398039 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.400514 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:27.400516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:27.400516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:27.400523 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:27.402191 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.402523 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.402524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.402524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.402575 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.402781 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.402782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.402782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.402901 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:27.403068 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:27.403111 (+ 43us) server_negotiation.cc:299] Negotiation successful
0621 05:25:27.403203 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Jun 21 05:25:27 dist-test-slave-b357 krb5kdc[24802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019527, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.60@KRBTEST.COM for kudu/127.2.32.60@KRBTEST.COM
Jun 21 05:25:27 dist-test-slave-b357 krb5kdc[24802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.32.61@KRBTEST.COM
I20260621 05:25:27.410006 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.401883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55400 (local address 127.2.32.60:34995)
0621 05:25:27.401918 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.401920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.401955 (+ 35us) server_negotiation.cc:407] Connection header received
0621 05:25:27.402015 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.402015 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.402024 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.402090 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.402353 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.403074 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.404043 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.404326 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.404383 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.406563 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:27.406564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:27.406564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:27.406569 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:27.408496 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.408845 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.408846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.408846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.408890 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.409053 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.409054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.409054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.409218 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:27.409291 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:27.409796 (+ 505us) server_negotiation.cc:299] Negotiation successful
0621 05:25:27.409922 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":17}
I20260621 05:25:27.410247 25037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.401843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:34995 (local address 127.0.0.1:55400)
0621 05:25:27.401915 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.401925 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 05:25:27.401966 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:25:27.402096 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:25:27.402097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:25:27.402241 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:25:27.402275 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:27.402276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.403184 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.403185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:27.403956 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:27.403959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.404423 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.404423 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:27.404478 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.405006 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:25:27.405010 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:25:27.406475 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:25:27.408586 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:27.408586 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:27.408587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:27.408777 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:27.408947 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:27.408947 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:27.408948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:27.408996 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:27.409604 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:25:27.409606 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:25:27.409656 (+ 50us) client_negotiation.cc:769] Sending connection context
0621 05:25:27.409750 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 05:25:27.409856 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260621 05:25:27.412060 25034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92: Bootstrap starting.
I20260621 05:25:27.412254 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.403787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38396 (local address 127.2.32.61:42369)
0621 05:25:27.403818 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.403820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.403827 (+ 7us) server_negotiation.cc:407] Connection header received
0621 05:25:27.403879 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.403880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.403889 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.403931 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.404377 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.405098 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.406111 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.406417 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.406503 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.409092 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:27.409093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:27.409094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:27.409100 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:27.411004 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.411411 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.411412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.411412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.411475 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.411698 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.411699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.411699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.411842 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:27.412042 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:27.412090 (+ 48us) server_negotiation.cc:299] Negotiation successful
0621 05:25:27.412170 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260621 05:25:27.412868 25034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:27.413230 25034 log.cc:824] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:27.413635 2176 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 05:25:27.414531 24857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8f7054764766468c8fc78671b52824a0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37044
I20260621 05:25:27.414654 24857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260621 05:25:27.414687 24857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:27.414800 25034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92: No bootstrap required, opened a new log
I20260621 05:25:27.415515 24857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7054764766468c8fc78671b52824a0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } } peers { permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } } peers { permanent_uuid: "951c5567bfc444d3961766f841f92b92" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } }
I20260621 05:25:27.415731 24857 leader_election.cc:288] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [CANDIDATE]: Term 1 election: Requested vote from peers a1fd9327993f438d841f13ab42b77bab (127.2.32.61:42369), 951c5567bfc444d3961766f841f92b92 (127.2.32.60:34995)
I20260621 05:25:27.416771 25034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7054764766468c8fc78671b52824a0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } } peers { permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } } peers { permanent_uuid: "951c5567bfc444d3961766f841f92b92" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } }
I20260621 05:25:27.416905 25034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951c5567bfc444d3961766f841f92b92, State: Initialized, Role: FOLLOWER
I20260621 05:25:27.416985 25034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7054764766468c8fc78671b52824a0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } } peers { permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } } peers { permanent_uuid: "951c5567bfc444d3961766f841f92b92" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } }
I20260621 05:25:27.417250 25034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:27.417229 25059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7054764766468c8fc78671b52824a0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } } peers { permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } } peers { permanent_uuid: "951c5567bfc444d3961766f841f92b92" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } } }
I20260621 05:25:27.417495 25059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:25:27.417826 25008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f7054764766468c8fc78671b52824a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "951c5567bfc444d3961766f841f92b92"
I20260621 05:25:27.417941 25008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92 [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:27.419365 25008 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f7054764766468c8fc78671b52824a0 in term 1.
I20260621 05:25:27.419601 24831 leader_election.cc:302] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f7054764766468c8fc78671b52824a0, 951c5567bfc444d3961766f841f92b92; no voters:
I20260621 05:25:27.419730 25041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 05:25:27.419868 25041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [term 1 LEADER]: Becoming Leader. State: Replica: 8f7054764766468c8fc78671b52824a0, State: Running, Role: LEADER
I20260621 05:25:27.419947 25041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7054764766468c8fc78671b52824a0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } } peers { permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } } peers { permanent_uuid: "951c5567bfc444d3961766f841f92b92" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } }
I20260621 05:25:27.420333 25041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f7054764766468c8fc78671b52824a0. Latest consensus state: current_term: 1 leader_uuid: "8f7054764766468c8fc78671b52824a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7054764766468c8fc78671b52824a0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } } peers { permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } } peers { permanent_uuid: "951c5567bfc444d3961766f841f92b92" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } } }
I20260621 05:25:27.420405 25041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:27.420857 25076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 05:25:27.421082 25076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260621 05:25:27.422686 25074 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:25:27.422737 25074 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:25:27.423740 24958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } attempt: 3
I20260621 05:25:27.422040 24965 master_runner.cc:186] Error getting master registration for 127.2.32.61:42369: OK, instance_id { permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" instance_seqno: 1782019527186096 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 05:25:27.424386 24965 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 05:25:27.425475 25008 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92 [term 1 FOLLOWER]: Refusing update from remote peer 8f7054764766468c8fc78671b52824a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:25:27.426191 25073 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "951c5567bfc444d3961766f841f92b92" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:25:27.427304 25059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f7054764766468c8fc78671b52824a0. Latest consensus state: current_term: 1 leader_uuid: "8f7054764766468c8fc78671b52824a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7054764766468c8fc78671b52824a0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } } peers { permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } } peers { permanent_uuid: "951c5567bfc444d3961766f841f92b92" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } } }
I20260621 05:25:27.427412 25059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92 [sys.catalog]: This master's current role is: FOLLOWER
Jun 21 05:25:27 dist-test-slave-b357 krb5kdc[24802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782019527, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.32.61@KRBTEST.COM for kudu/127.2.32.60@KRBTEST.COM
I20260621 05:25:27.432719 24960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.424005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.32.60:34995 (local address 127.0.0.1:55424)
0621 05:25:27.424051 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0621 05:25:27.424060 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 05:25:27.424092 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 05:25:27.424242 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 05:25:27.424242 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 05:25:27.424405 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0621 05:25:27.424447 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:27.424449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.425351 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.425352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:27.426034 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 05:25:27.426036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.427482 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.427483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 05:25:27.427545 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.428109 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 05:25:27.428114 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 05:25:27.429652 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 05:25:27.431635 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:27.431636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:27.431637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:27.431845 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:27.432052 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 05:25:27.432053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 05:25:27.432053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0621 05:25:27.432111 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 05:25:27.432386 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 05:25:27.432387 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 05:25:27.432433 (+ 46us) client_negotiation.cc:769] Sending connection context
0621 05:25:27.432508 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 05:25:27.432610 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260621 05:25:27.433264 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.424057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.2.32.60:34995)
0621 05:25:27.424091 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.424092 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.424100 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:25:27.424154 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.424155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.424164 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.424208 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.424525 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.425267 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.426123 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.426425 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.426496 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.429747 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:27.429749 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:27.429749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:27.429756 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:27.431540 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.431924 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.431925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.431925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.431984 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.432176 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.432177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.432177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.432302 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:27.432360 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:27.433077 (+ 717us) server_negotiation.cc:299] Negotiation successful
0621 05:25:27.433163 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260621 05:25:27.433878 25073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f7054764766468c8fc78671b52824a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7054764766468c8fc78671b52824a0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } } peers { permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } } peers { permanent_uuid: "951c5567bfc444d3961766f841f92b92" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } } }
I20260621 05:25:27.433998 25073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:27.434293 25076 catalog_manager.cc:1347] Generated new cluster ID: d4b94da022124cf8bab951dc058f2b00
I20260621 05:25:27.434338 25076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 05:25:27.434535 25059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f7054764766468c8fc78671b52824a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7054764766468c8fc78671b52824a0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } } peers { permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } } peers { permanent_uuid: "951c5567bfc444d3961766f841f92b92" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } } }
I20260621 05:25:27.434609 25059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:25:27.435842 24958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab: Bootstrap starting.
I20260621 05:25:27.436510 24958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab: Neither blocks nor log segments found. Creating new log.
I20260621 05:25:27.436811 24958 log.cc:824] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab: Log is configured to *not* fsync() on all Append() calls
I20260621 05:25:27.437120 25079 mvcc.cc:205] Tried to move back new op lower bound from 7299151984332029952 to 7299151984312705024. Current Snapshot: MvccSnapshot[applied={T|T < 7299151984332029952}]
I20260621 05:25:27.440126 24958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab: No bootstrap required, opened a new log
I20260621 05:25:27.441989 24958 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7054764766468c8fc78671b52824a0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } } peers { permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } } peers { permanent_uuid: "951c5567bfc444d3961766f841f92b92" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } }
I20260621 05:25:27.442127 24958 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1fd9327993f438d841f13ab42b77bab, State: Initialized, Role: FOLLOWER
I20260621 05:25:27.442210 24958 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7054764766468c8fc78671b52824a0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } } peers { permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } } peers { permanent_uuid: "951c5567bfc444d3961766f841f92b92" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } }
I20260621 05:25:27.442520 24958 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab [sys.catalog]: configured and running, proceeding with master startup.
I20260621 05:25:27.442565 24932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f7054764766468c8fc78671b52824a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a1fd9327993f438d841f13ab42b77bab"
I20260621 05:25:27.442643 24931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab [term 0 FOLLOWER]: Advancing to term 1
I20260621 05:25:27.443578 24931 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab [term 1 FOLLOWER]: Refusing update from remote peer 8f7054764766468c8fc78671b52824a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 05:25:27.443897 25041 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 05:25:27.445259 24932 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f7054764766468c8fc78671b52824a0 in term 1.
I20260621 05:25:27.449132 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.417354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38398 (local address 127.2.32.61:42369)
0621 05:25:27.417663 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.417665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.419586 (+ 1921us) server_negotiation.cc:407] Connection header received
0621 05:25:27.420193 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.420194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.420205 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.420236 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.421948 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.422749 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.428947 (+ 6198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.429278 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.429324 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.437538 (+ 8214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:27.437540 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:27.437540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:27.437549 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:27.439517 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.448282 (+ 8765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.448283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.448283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.448353 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.448548 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.448549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.448549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.448680 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:27.448764 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:27.448956 (+ 192us) server_negotiation.cc:299] Negotiation successful
0621 05:25:27.449037 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281}
W20260621 05:25:27.451683 25095 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 05:25:27.451737 25095 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 05:25:27.453203 24887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.417084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37052 (local address 127.2.32.62:38777)
0621 05:25:27.417619 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.417621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.419574 (+ 1953us) server_negotiation.cc:407] Connection header received
0621 05:25:27.419598 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.419600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.419611 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.419663 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.422854 (+ 3191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.423554 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.426814 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.427325 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.427427 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.439688 (+ 12261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:27.439690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:27.439690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:27.439700 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:27.441708 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.450209 (+ 8501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.450210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.450211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.450274 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.450466 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.450467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.450467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.450586 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:27.450672 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:27.453024 (+ 2352us) server_negotiation.cc:299] Negotiation successful
0621 05:25:27.453114 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511}
I20260621 05:25:27.454622 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:27.417768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.2.32.60:34995)
0621 05:25:27.417802 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.417804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.422036 (+ 4232us) server_negotiation.cc:407] Connection header received
0621 05:25:27.422122 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.422123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.422133 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.422161 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.425406 (+ 3245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.426088 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.432360 (+ 6272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:27.432680 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:27.432749 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:27.450732 (+ 17983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 05:25:27.450733 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 05:25:27.450733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 05:25:27.450742 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 05:25:27.452504 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.453797 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.453798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.453798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.453857 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 05:25:27.454066 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 05:25:27.454067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 05:25:27.454067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 05:25:27.454200 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 05:25:27.454274 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:27.454450 (+ 176us) server_negotiation.cc:299] Negotiation successful
0621 05:25:27.454528 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260621 05:25:27.455189 25084 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f7054764766468c8fc78671b52824a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7054764766468c8fc78671b52824a0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } } peers { permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } } peers { permanent_uuid: "951c5567bfc444d3961766f841f92b92" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } } }
I20260621 05:25:27.455279 25084 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:25:27.455242 25099 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f7054764766468c8fc78671b52824a0. Latest consensus state: current_term: 1 leader_uuid: "8f7054764766468c8fc78671b52824a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7054764766468c8fc78671b52824a0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } } peers { permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } } peers { permanent_uuid: "951c5567bfc444d3961766f841f92b92" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } } }
I20260621 05:25:27.455325 25099 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:25:27.455798 25076 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 05:25:27.456343 25101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f7054764766468c8fc78671b52824a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7054764766468c8fc78671b52824a0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } } peers { permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } } peers { permanent_uuid: "951c5567bfc444d3961766f841f92b92" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } } }
I20260621 05:25:27.456494 25076 catalog_manager.cc:1504] Loading token signing keys...
I20260621 05:25:27.456499 25101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0 [sys.catalog]: This master's current role is: LEADER
I20260621 05:25:27.455330 25084 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f7054764766468c8fc78671b52824a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7054764766468c8fc78671b52824a0" member_type: VOTER last_known_addr { host: "127.2.32.62" port: 38777 } } peers { permanent_uuid: "a1fd9327993f438d841f13ab42b77bab" member_type: VOTER last_known_addr { host: "127.2.32.61" port: 42369 } } peers { permanent_uuid: "951c5567bfc444d3961766f841f92b92" member_type: VOTER last_known_addr { host: "127.2.32.60" port: 34995 } } }
I20260621 05:25:27.459328 25084 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab [sys.catalog]: This master's current role is: FOLLOWER
I20260621 05:25:27.456992 25097 mvcc.cc:205] Tried to move back new op lower bound from 7299151984332029952 to 7299151984312705024. Current Snapshot: MvccSnapshot[applied={T|T < 7299151984332029952}]
I20260621 05:25:27.463513 25076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f7054764766468c8fc78671b52824a0: Generated new TSK 0
I20260621 05:25:27.463672 25076 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 05:25:27.491814 24887 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:27.490024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37058 (local address 127.2.32.62:38777)
0621 05:25:27.490071 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.490073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.490953 (+ 880us) server_negotiation.cc:407] Connection header received
0621 05:25:27.491061 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.491062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.491073 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.491116 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.491743 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37058: BlockingRecv error: recv got EOF from 127.0.0.1:37058 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260621 05:25:27.492543 24962 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:27.490414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38412 (local address 127.2.32.61:42369)
0621 05:25:27.490453 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.490455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.492042 (+ 1587us) server_negotiation.cc:407] Connection header received
0621 05:25:27.492151 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.492152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.492161 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.492190 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.492484 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38412: BlockingRecv error: recv got EOF from 127.0.0.1:38412 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260621 05:25:27.493245 25039 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:27.490703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55438 (local address 127.2.32.60:34995)
0621 05:25:27.490742 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.490744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.492762 (+ 2018us) server_negotiation.cc:407] Connection header received
0621 05:25:27.492855 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.492856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.492866 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.492894 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.493183 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55438: BlockingRecv error: recv got EOF from 127.0.0.1:55438 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260621 05:25:27.495517 24962 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:27.494709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.2.32.61:42369)
0621 05:25:27.494741 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.494743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.494906 (+ 163us) server_negotiation.cc:407] Connection header received
0621 05:25:27.495043 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.495044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.495060 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.495089 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.495475 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38416: BlockingRecv error: recv got EOF from 127.0.0.1:38416 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260621 05:25:27.498402 24962 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:27.497687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38424 (local address 127.2.32.61:42369)
0621 05:25:27.497718 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.497720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.497911 (+ 191us) server_negotiation.cc:407] Connection header received
0621 05:25:27.498043 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.498043 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.498053 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.498080 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.498361 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38424: BlockingRecv error: recv got EOF from 127.0.0.1:38424 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260621 05:25:27.502101 24962 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:27.501442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38430 (local address 127.2.32.61:42369)
0621 05:25:27.501471 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.501473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.501658 (+ 185us) server_negotiation.cc:407] Connection header received
0621 05:25:27.501742 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.501743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.501752 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.501779 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.502053 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38430: BlockingRecv error: recv got EOF from 127.0.0.1:38430 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260621 05:25:27.507999 24962 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:27.507277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38444 (local address 127.2.32.61:42369)
0621 05:25:27.507311 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.507312 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.507514 (+ 202us) server_negotiation.cc:407] Connection header received
0621 05:25:27.507608 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.507609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.507620 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.507647 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.507958 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38444: BlockingRecv error: recv got EOF from 127.0.0.1:38444 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260621 05:25:27.518049 24962 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:27.517212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38446 (local address 127.2.32.61:42369)
0621 05:25:27.517245 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.517247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.517456 (+ 209us) server_negotiation.cc:407] Connection header received
0621 05:25:27.517589 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.517590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.517601 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.517629 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.517982 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38446: BlockingRecv error: recv got EOF from 127.0.0.1:38446 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260621 05:25:27.536759 24962 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:27.535608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38460 (local address 127.2.32.61:42369)
0621 05:25:27.535667 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.535676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.535941 (+ 265us) server_negotiation.cc:407] Connection header received
0621 05:25:27.536260 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.536262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.536276 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.536307 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.536701 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38460: BlockingRecv error: recv got EOF from 127.0.0.1:38460 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260621 05:25:27.571086 24962 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:27.570165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38466 (local address 127.2.32.61:42369)
0621 05:25:27.570208 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.570211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.570497 (+ 286us) server_negotiation.cc:407] Connection header received
0621 05:25:27.570610 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.570612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.570624 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.570653 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.571022 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38466: BlockingRecv error: recv got EOF from 127.0.0.1:38466 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260621 05:25:27.637475 24962 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:27.636725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38470 (local address 127.2.32.61:42369)
0621 05:25:27.636775 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.636777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.636917 (+ 140us) server_negotiation.cc:407] Connection header received
0621 05:25:27.636944 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.636945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.636955 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.637007 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.637373 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38470: BlockingRecv error: recv got EOF from 127.0.0.1:38470 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260621 05:25:27.767956 24962 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:27.766897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38484 (local address 127.2.32.61:42369)
0621 05:25:27.766970 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:27.766973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:27.767066 (+ 93us) server_negotiation.cc:407] Connection header received
0621 05:25:27.767114 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:27.767115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:27.767170 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:27.767236 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:27.767870 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38484: BlockingRecv error: recv got EOF from 127.0.0.1:38484 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260621 05:25:28.026290 24962 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 05:25:28.025550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38500 (local address 127.2.32.61:42369)
0621 05:25:28.025624 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:28.025626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:28.025636 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:25:28.025690 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:28.025691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:28.025700 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:28.025751 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 05:25:28.026201 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38500: BlockingRecv error: recv got EOF from 127.0.0.1:38500 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 05:25:28.423211 25074 catalog_manager.cc:1259] Loaded cluster ID: d4b94da022124cf8bab951dc058f2b00
I20260621 05:25:28.423295 25074 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92: loading cluster ID for follower catalog manager: success
I20260621 05:25:28.424063 25074 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92: acquiring CA information for follower catalog manager: success
I20260621 05:25:28.424302 25074 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 951c5567bfc444d3961766f841f92b92: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:25:28.452229 25095 catalog_manager.cc:1259] Loaded cluster ID: d4b94da022124cf8bab951dc058f2b00
I20260621 05:25:28.452314 25095 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab: loading cluster ID for follower catalog manager: success
I20260621 05:25:28.453080 25095 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab: acquiring CA information for follower catalog manager: success
I20260621 05:25:28.453352 25095 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a1fd9327993f438d841f13ab42b77bab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 05:25:28.542701 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:28.540022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.2.32.61:42369)
0621 05:25:28.540154 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:28.540156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:28.540166 (+ 10us) server_negotiation.cc:407] Connection header received
0621 05:25:28.540242 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:28.540242 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:28.540292 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:28.540354 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:25:28.540504 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:28.540998 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:28.541824 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:28.542076 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:28.542132 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:28.542284 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:25:28.542388 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:25:28.542487 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:28.542589 (+ 102us) server_negotiation.cc:299] Negotiation successful
0621 05:25:28.542607 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260621 05:25:28.547693 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:28.544767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52980 (local address 127.2.32.60:34995)
0621 05:25:28.544918 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:28.544919 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:28.544928 (+ 9us) server_negotiation.cc:407] Connection header received
0621 05:25:28.544973 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:28.544973 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:28.545007 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:28.545085 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:25:28.545172 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:28.545723 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:28.546525 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:28.546778 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:28.546883 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:28.547119 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:25:28.547380 (+ 261us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:25:28.547497 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:28.547575 (+ 78us) server_negotiation.cc:299] Negotiation successful
0621 05:25:28.547601 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":111,"threads_started":1}
I20260621 05:25:28.552984 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:28.550272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52982 (local address 127.2.32.60:34995)
0621 05:25:28.550325 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:28.550327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:28.550432 (+ 105us) server_negotiation.cc:407] Connection header received
0621 05:25:28.550476 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:28.550477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:28.550489 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:28.550532 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:25:28.550689 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:28.551463 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:28.552297 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:28.552541 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:28.552613 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:28.552721 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:25:28.552754 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:25:28.552802 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:28.552852 (+ 50us) server_negotiation.cc:299] Negotiation successful
0621 05:25:28.552875 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260621 05:25:28.553602 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:28.550006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38288 (local address 127.2.32.61:42369)
0621 05:25:28.550057 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:28.550059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:28.550263 (+ 204us) server_negotiation.cc:407] Connection header received
0621 05:25:28.550360 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:28.550364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:28.550376 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:28.550445 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:25:28.550713 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:28.551367 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:28.552553 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:28.552880 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:28.552955 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:28.553191 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:25:28.553228 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:25:28.553274 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:28.553521 (+ 247us) server_negotiation.cc:299] Negotiation successful
0621 05:25:28.553542 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 05:25:28.554143 25181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 05:25:28.549988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58584 (local address 127.2.32.62:38777)
0621 05:25:28.550263 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0621 05:25:28.550266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 05:25:28.550274 (+ 8us) server_negotiation.cc:407] Connection header received
0621 05:25:28.550334 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 05:25:28.550335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 05:25:28.550406 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 05:25:28.550525 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 05:25:28.550632 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:28.551493 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:28.552996 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 05:25:28.553281 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 05:25:28.553521 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 05:25:28.553593 (+ 72us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 05:25:28.553722 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 05:25:28.553787 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 05:25:28.554012 (+ 225us) server_negotiation.cc:299] Negotiation successful
0621 05:25:28.554041 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":65,"threads_started":1}
I20260621 05:25:28.559690 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 24815
I20260621 05:25:28.565217 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 24890
I20260621 05:25:28.570444 2176 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJjmGzv/build/release/bin/kudu with pid 24965
2026-06-21T05:25:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2869 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5727 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (303466 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS